Generation of three-dimensional scans for intraoperative imaging

ABSTRACT

A system for executing a three-dimensional (3D) intraoperative scan of a patient is disclosed. A 3D scanner controller projects the object points included onto a first image plane and the object points onto a second image plane. The 3D scanner controller determines first epipolar lines associated with the first image plane and second epipolar lines associated with the second image plane based on an epipolar plane that triangulates the object points included in the first 2D intraoperative image to the object points included in the second 2D intraoperative image. Each epipolar lines provides a depth of each object as projected onto the first image plane and the second image plane. The 3D scanner controller converts the first 2D intraoperative image and the second 2D intraoperative image to the 3D intraoperative scan of the patient based on the depth of each object point provided by each corresponding epipolar line.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/951,480 filed on Dec. 20, 2019 and U.S. Provisional Application No.63,040,816 filed on Jun. 18, 2020, which are incorporated herein byreference in their entirety.

TECHNICAL FIELD

Generally, the present invention relates to apparatuses, systems, andmethods of surgical imaging, navigation and tracking. In particular, thepresent invention relates to systems and methods for performing 3Dscanning, surgical imaging, tracking, image processing, computer vision,image registration, and display. More particularly, the presentinvention relates to an apparatus, system, and method for utilizing 3Dscanning, intraoperative imaging, light source, tracking hardware, incombination with image processing and computer vision algorithms, toperform procedural guidance.

BACKGROUND OF THE INVENTION

Current surgical imaging and navigation hardware and software, such asthose used in the spine and orthopedic fields, still fail to deliverrobust procedure guidance, as desired by surgeons. There is a need for asystem that can provide accurate guidance for surgical applications forhard tissues and soft tissues alike. There is a need for systems that iscapable of being used in various applications such as surgery,therapeutic monitoring, and medical training. Furthermore, there is aneed for an imaging system that combines augmented reality, real timeimaging, procedure guidance, and decision support.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

Embodiments of the present disclosure are described with reference tothe accompanying drawings. In the drawings, like reference numeralsindicate identical or functionally similar elements. Additionally, theleft most digit(s) of a reference number typically identifies thedrawing in which the reference number first appears.

FIG. 1 illustrates a block diagram of a surgical imaging and navigationsystem;

FIG. 2 is a schematic view of a 3D scanner epipolar geometryconfiguration where the 3D scanner implements epipolar geometry;

FIG. 3 is a schematic view of random patterns that the statisticalpattern generator may generate;

FIG. 4 illustrates a schematic view of dynamic patterns that thenon-statistical pattern generator generates such as binary code, stripeboundary code, and miere pattern;

FIG. 5 illustrates a schematic view of defocusing a binary pattern inthe defocusing configuration;

FIG. 6A illustrates a schematic view of an example of U-netarchitecture;

FIG. 6B illustrates a schematic view of an example of GAN configuration;

FIG. 7 illustrates a block diagram of a master-slave system with morethan one 3D scanners;

FIG. 8 illustrates a block diagram of a 3D scanning controller and a 3Dscanner that interacts with a surgical tool;

FIG. 9 illustrates a schematic view of a 3D scanner that is situated atthe end of the smart surgical tool closer to the patient and the displayis situated at the end of the smart surgical tool closer theuser/surgeon;

FIG. 10 illustrates a block diagram of a 3D scanning configuration thatincludes a communication interface that communicates with othercomputers or surgical navigation systems.

DETAILED DESCRIPTION OF THE INVENTION

A surgical imaging and navigation system 100 of the present invention isshown in FIG. 1. In one embodiment, the surgical imaging and navigationsystem 100 includes a display 120, a controller 110, a 3D scanningcontroller 150, a 3D scanner 160, an intraoperative imaging controller170, an imaging detector 180, a light source controller 190, a lightsource 195, a tracking controller 140, and a tracker 130. The 3Dscanning controller 150 controls the modes and properties of the 3Dscanner 160. For instance, the size of the area of 3D scanning, theresolution of 3D scanning, the speed of 3D scanning, the timing of 3Dscanning may be controlled by the 3D scanning controller 150. Theintraoperative imaging controller 170 controls the modes and propertiesof imaging detector 180. For instance, the size of the area ofintraoperative imaging, the resolution of intraoperative imaging, thespeed of intraoperative imaging, the timing of intraoperative imaging,and the mode of intraoperative imaging may be controlled by theintraoperative imaging controller 170.

The light source controller 190 controls the modes and properties oflight source. For instance, the size of the area of the light source 195to shine on, the power of the light source 195, the wavelength of thelight source 195, the frequency of the light source 195, the timing ofthe light source 195 may be controlled by the light source controller190. The tracker 130 may track the surgical tools and other objects, viaoptical tracking, electromagnetic tracking, or a combination thereof.The tracking controller 140 controls how the tracker 130 tracks thesurgical tools and other objects. The display 120 may display thesurgical navigation information to the user. The controller 110 is in inoperative communication with the 3D scanning controller 150,intraoperative imaging controller 170, light source controller 190, thetracking controller 140, and the display 120. The controller may runsoftware such as image registration software or computer visionalgorithms to enable surgical navigation. The display 120 may be displaymedical information to the user in 2D or 3D. For example, the display120 may be a traditional 2D monitor or a head mounted display that maydisplay images to the user in 3D. It should be appreciated thedescriptions above are only one example how the surgical imaging andnavigation system 100 may work.

With an exemplary system previously discussed, surgical imaging andnavigation may be implemented to provide intraoperative guidance tosurgeons and other medical professionals. The 3D scanner 160 may capturea 3D scan of an anatomy of a patient as controlled by the 3D scanningcontroller 150. The imaging detector 180 may capture at least one modeof an intraoperative image of a patient as controlled by theintraoperative imaging controller 170. The light source 195 may provideintraoperative illumination on the patient as controlled by the lightsource controller 190. The controller 110 may register the preoperativeimage data to the intraoperative 3D scan. The tracker 130 may track atleast one entity in surgery as controlled by the tracking controller140. The display 120 may display the surgical navigation information tothe user.

3D Scanner with Epipolar Geometry

FIG. 1 illustrates a block diagram of the surgical imaging andnavigation system 100 that may execute a 3D intraoperative scan of apatient to generate a plurality of intraoperative images of the patientthat enables a surgeon to navigate a surgical operation on the patient.The 3D scanner 160 that includes a first image sensor 205 a and a secondimage sensor 205 n may capture a first two-dimensional (2D)intraoperative image 260 a of a plurality of object points 250(a-n),where n is an integer equal to or greater than one and is associatedwith the patient via the first image sensor 205 a and a second 2Dintraoperative image 265 n of the plurality of object points 250(a-n)via the second image sensor 205 n. For example, FIG. 2 is a schematicview of a 3D scanner epipolar geometry configuration 200 where the 3Dscanner 160 implements epipolar geometry.

The 3D scanning controller 150 may project the plurality of objectpoints 250(a-n) included in the first 2D intraoperative image 260 a ontoa first image plane 265 a associated with the first image sensor 205 aand the plurality of object points 250(a-n) included in the second 2Dintraoperative image 260 n onto a second image plane 265 n associatedwith the second image sensor 205 n. The 3D scanning controller 150 maydetermine a plurality of first epipolar lines 220 a associated with thefirst image plane 265 a and a plurality of second epipolar lines 220 nassociated with the second image plane 265 n based on an epipolar plane210 that triangulates the plurality of object points 250(a-n) includedin the first 2D intraoperative image 260 a to the plurality of objectpoints 250(a-n) included in the second 2D intraoperative image 260 n.Each epipolar line 220(a-n) provides depth of each object point 250(a-n)as projected onto the first image plane 265 a associated with the firstimage sensor 205 a and the second image plane 265 n associated with thesecond image sensor 205 n. The 3D scanning controller 150 may convertthe first 2D intraoperative image 260 a and the second 2D intraoperativeimage 260 n to the 3D intraoperative scan of the patient based on thedepth of each object point 250(a-n) provided by each correspondingepipolar line 220(a-n).

The controller 110 may co-register pre-operative image data capturedfrom at least one pre-operative image of the patient with intraoperativeimage data provided by the 3D intraoperative scan. The controller 110may instruct the display 120 to display the co-registered pre-operativeimage data as captured by the at least one pre-operative image with theintraoperative image data provided by the 3D intraoperative scan as thesurgeon navigates during the surgical operation.

The 3D scanner 160 may be controlled by the 3D scanning controller 150to perform a 3D scan of the patient during the surgery. In oneembodiment, the 3D scanner 160 includes two or more image sensors260(a-n) and two or more lenses for 3D scanning. An example of 3Dscanning for surgical navigation using an epipolar geometryconfiguration includes two image sensors 205(a-n). An example of 3Dscanning for surgical navigation using an epipolar geometryconfiguration includes a projector and an image sensor. P is a point ina 3D space, p^(l) and p^(r) are the corresponding object points 250(a-n)on two 2D imaging planes 265(a-n). The focal points of the left lenso^(l), the focal points of the right lens or and the object point pforms a plane called epipolar plane 210. The intersection between theepipolar plane 210 and left imaging plane 265 a is a line called leftepipolar line L^(l) 220 a. The intersection between the epipolar plane210 and the right imaging plane 265 n is a line called right epipolarline L^(r) 220 n and e ^(l) is the left epipole 270 a and e ^(r) is theright epipole 270 n.

The 3D scanning controller 150 may generate the plurality of firstepipolar lines 220 a positioned in the first image plane 265 a of thefirst 2D intraoperative image 265 a. In one aspect, after systemcalibration, rectification, and un-distortion, each of the firstepipolar lines 220 a is parallel to each other as positioned in thefirst image plane 265 a. The 3D scanning controller 150 may generate theplurality of second epipolar lines 220 n positioned in the second imageplane 265 n of the second 2D intraoperative image 265 n. Each of thesecond epipolar lines 220 n is parallel to each other second epipolarlines 220 n as positioned in the second image plane 265 n. The 3Dscanning controller 150 may convert the first 2D intraoperative image265 a and the second 2D intraoperative image 265 n to the 3Dintraoperative scan of the patient based on the depth of each objectpoint 250(a-n) provided by each corresponding first epipolar line 220 aand second epipolar line 220 n as positioned in the corresponding firstimage plane 265 a and second image plane 265 n.

The 3D scanning controller 150 may conjugate each first epipolar line220 a positioned in the first imaging plane 265 a of the first 2Dintraoperative image 260 a to each corresponding second epipolar line220 n positioned in the second image plane 265 n of the second 2Dintraoperative image 260 n. The plurality of first epipolar lines 220 aand the plurality of second epipolar lines may be a conjugate set. The3D scanning controller 150 may convert the first 2D intraoperative image260 a and the second 2D intraoperative image 260 n to the 3Dintraoperative scan of the patient based on the depth of each objectpoint 250(a-n) provided by each corresponding conjugate of each other aspositioned in the corresponding first image plane 265 a and the secondimage plane 265 n. The search for corresponding image points 265(a-n)between the first 2D intraoperative image 260 a and the second 2Dintraoperative image 260 n is conducted on a first epipolar line 220 aand a second epipolar line 220 n.

The 3D scanning controller 150 may generate each first epipolar line 220a positioned in the first imaging plane 265 a of the first 2Dintraoperative image 260 a to correspond to a set of first pixelsincluded in the first 2D intraoperative image 260 a. The 3D scanningcontroller 150 may generate each second epipolar line 220 n positionedin the second image plane 265 n of the second 2D intraoperative image260 to correspond to a set of second pixels included in the second 2Dintraoperative image 260 n. The 3D scanning controller 150 may convertthe first 2D intraoperative image 260 a and the second 2D intraoperativeimage 260 n to the 3D intraoperative scan of the patient based on thedepth of each set of first pixels and for each corresponding firstepipolar line 220 a and the depth of each set of second pixels for eachcorresponding second epipolar line 220 n as positioned in the firstimage plane 265 a and the second image plane 265 n.

The 3D scanning controller 150 may generate each first epipolar line 220a positioned in the first image plane 265 a of the first 2Dintraoperative image 260 a to correspond to a row of first pixelsincluded in the first 2D intraoperative image 260 a. The 3D scanningcontroller 150 may generate each second epipolar line 220 n positionedin the second image plane 265 n to correspond to a row of second pixelsincluded in the second 2D intraoperative image 260 n. The 3D scanningcontroller 150 may convert the first 2D intraoperative image 260 a andthe second 2D intraoperative image 265 n to the 3D intraoperative imagescan of the patient based on the depth of each row of first pixels foreach corresponding first epipolar line 220 a and the depth of each rowof second pixels for each corresponding second epipolar line 220 n inthe first image plane 265 a and the second image plane 265 n.

The 3D scanning controller 150 may conduct a one-dimensional (1D) searchfor a corresponding pair of object points 250(a-n) on the first epipolarline 220 a of the first image plane 265 a of the first 2D intraoperativeimage 260 a and the second epipolar line 220 n in the second image plane265 n of the second 2D intraoperative image 265 n. The first objectpoint 250(a-n) positioned on the first epipolar line 220 a correspondsto a second object point 250(a-n) positioned on the second epipolar line220 n. The 3D scanning controller 150 may convert the 1D search of thecorresponding pair of object points 250(a-n) on the first epipolar line220 a and the second epipolar line 220 n to the 3D intraoperative scanof the patient based on the depth of the first object point 250(a-n) onthe first epipolar line 220 a and the corresponding second object point250(a-n) on the second epipolar line 220 n as positioned in the firstimage plane 265 a and the second image plane 265 n. In another example,the 3D scanner 160 searches for the corresponding object point pair onlyon the left epipolar line L^(l) 220 and the right epipolar line L^(r)220 n. This one-dimensional search can make the 3D scanning faster thanconventional methods that conduct exhaustive search, therebyaccelerating surgical navigation. In an embodiment, the 3D scanningcontroller 150 may conduct a windowed 2D search for a corresponding pairof object points 250(a-n) on the first epipolar line 220 a of the firstimage plane 265 a of the first 2D intraoperative image 260 a and thesecond epipolar line 220 n in the second image plane 265 n of the second2D intraoperative image 265 n.

In another example, the image sensors 205(a-n) used are high speedcomplementary metal-oxide-semiconductor (CMOS) image sensors. This makethe 3D scanning process very fast. For instance, the frame rate of 3Dscanning may be beyond 100 frame per second, up to 4000 frame persecond. In another example, the 3D scanner 160 includes two or moreimage sensors 205(a-n), two or more lenses, and a pattern generator 165.The pattern generator 165 may generate patterns to be projected on thepatients to be scanned. It is advantageous to create patterns via lighton the patient to be imaged by the image sensors 205(a-n). The patternscan help improve the robustness and accuracy of the 3D scan andtherefore improve surgical navigation.

In one example, the pattern generator 165 includes one or more lightemitting diodes (LEDs) and a patterned aperture. The patterned aperturemay be made of metals, ceramics or plastics. The patterned apertureswith the LEDs created patterns that may be combined with the informationof the patient anatomy, which increases the accuracy and speed of the 3Dscanning. The patterned aperture and epipolar geometry combined canfacilitate more accurate scanning of patient anatomy. Improved 3D scancan enhance the image registration between intraoperative 3D scan andpreoperative images (e.g. MRI and CT), thereby improving the surgicalnavigation. It should be appreciated that other illumination devicessuch as a halogen lamp, a xenon lamp, an arc lamp, a laser diode may beused instead of an LED.

In another example, the 3D scanner 160 includes or more image sensors205(a-n), two or more lenses, and a pattern generator 165 that includesone or more LEDs and a digital micromirror device. The LED and digitalmicromirror device may be controlled by the 3D scanning controller 150to create patterns desirable for the 3D scanning application inmedicine. The digital micromirror device with the LEDs created patternsthat may be projected on the patient anatomy, which increases theaccuracy and speed of the 3D scanning. The digital micromirror deviceand epipolar geometry combined can facilitate more accurate scanning ofpatient anatomy. Improved 3D scan can enhance the image registrationbetween intraoperative 3D scan and preoperative images (e.g. MRI andCT), thereby improving the surgical navigation. It should be appreciatedthat other illumination devices such as a halogen lamp, a xenon lamp, anarc lamp, a laser diode may be used instead of an LED. In one example,the patterns created Is dynamic where the pattern changes temporarily,so that any residual pattern-to-depth dependence may be reduced.

In yet another example, the 3D scanner 160 includes two or more imagesensors 205(a-n), two or more lenses, and a pattern generator 165 thatincludes one or more LEDs and a thin-film-transistor liquid-crystaldisplay. The LED and thin-film-transistor liquid-crystal display 120 maybe controlled by the 3D scanning controller 150 to generate patternsdesirable for the 3D scanning application in medicine. Thethin-film-transistor liquid-crystal display 120 with the LEDs generatedpatterns that may be projected on the patient anatomy, which increasesthe accuracy and speed of the 3D scanning. The thin-film-transistorliquid-crystal display 120 and epipolar geometry combined may facilitatemore accurate scanning of patient anatomy. Improved 3D scan may enhancethe image registration between intraoperative 3D scan and preoperativeimages (e.g. MRI and CT), thereby improving the surgical navigation. Itshould be appreciated that other illumination devices such as a halogenlamp, a xenon lamp, an arc lamp, a laser diode may be used instead of anLED. In one example, the patterns created may be dynamic where thepattern changes temporarily, so that any residual pattern-to-depthdependence may be reduced.

In yet another example, the 3D scanner 160 includes two or more imagesensors 205(a-n), two or more lenses, and a pattern generator 165 thatincludes one or more edge emitting laser, at least one collimating lens,and at least one diffractive optics element. The edge emitting laser andthe diffractive optics element may be controlled by the 3D scanningcontroller 150 to create patterns desirable for the 3D scanningapplication in medicine. An example of a pattern creator comprises anedge emitting laser, a collimating lens, and a diffractive opticselement. The edge emitting laser, the collimating lens and thediffractive optics element created patterns that may be projected on thepatient anatomy, which increases the accuracy and speed of the 3Dscanning. The edge emitting laser, the diffractive optics element andepipolar geometry combined may facilitate more accurate scanning ofpatient anatomy. An improved 3D scan may enhance the image registrationbetween intraoperative 3D scan and preoperative images (e.g. MRI andCT), thereby improving the surgical navigation. It should be appreciatedthat other illumination devices such as LEDs, a halogen lamp, a xenonlamp, an arc lamp, a laser diode may be used instead of an edge emittinglaser. In one example, the patterns created Is dynamic where the patternchanges temporarily, so that any residual pattern-to-depth dependencemay be reduced.

In another example, the 3D scanner comprises 2 or more image sensors, 2or more lenses, and a pattern creator that comprises at least onepatterned vertical cavity semiconductor emission laser array, at leastone collimating lens, and at least one diffractive optics element. Thepatterned vertical cavity semiconductor emission laser array and thediffractive optics element may be controlled by the 3D scanningcontroller to create patterns desirable for the 3D scanning applicationin medicine. An example of a pattern creator comprises a patternedvertical cavity semiconductor emission laser array, a collimating lens,and a diffractive optics element. The patterned vertical cavitysemiconductor emission laser array, the collimating lens and thediffractive optics element creates patterns that may be projected on thepatient anatomy, which increases the accuracy and speed of the 3Dscanning. The patterned vertical cavity semiconductor emission laserarray, the diffractive optics element and epipolar geometry combined canfacilitate more accurate scanning of patient anatomy. Improved 3D scancan enhance the image registration between intraoperative 3D scan andpreoperative images (e.g. MRI and CT), thereby improving the surgicalnavigation. In one example, the patterns created Is dynamic where thepattern changes temporarily, so that any residual pattern-to-depthdependence may be reduced.

In another embodiment, the 3D scanner includes two or moreinfrared-sensitive image sensors 205(a-n), two or moreinfrared-compatible lenses, and an infrared pattern generator 165. The3D scanner 160 may further include two or more infrared optical filters.In one example, the infrared optical filters are used in conjunctionwith infrared-sensitive image sensors 205(a-n) and infrared-compatiblelenses to capture infrared images 260(a-n). In one example, the infraredrange are beyond 800 nm. The infrared optical filters may be bandpassfilters or long-pass filters (e.g. 800 nm long pass filters or 830 nmband pass filters). In one aspect, the infrared-sensitive image sensors205(a-n) may be high speed infrared-sensitive CMOS image sensors205(a-n). In one example, the infrared pattern generator 165 includesone or more light emitting diodes (LEDs) and a patterned aperture. Inanother example, the infrared pattern generator 165 includes one or moreinfrared LEDs and an infrared-compatible digital micromirror device. Inyet another example, the infrared pattern generator 165 includes one ormore LEDs and an infrared-compatible thin-film-transistor liquid-crystaldisplay 120. In yet another example, the infrared pattern generator 165includes one or more infrared edge emitting laser, at least oneinfrared-compatible collimating lens, and at least one diffractiveoptics element. In yet another example, the infrared pattern generatorincludes at least one infrared patterned vertical cavity semiconductoremission laser array, at least one infrared-compatible collimating lens,and at least one diffractive optics element. It should be appreciatedthat other infrared illumination devices such as an infrared halogenlamp, an infrared xenon lamp, an infrared arc lamp, an infrared laserdiode may be used. In one example, the infrared patterns created aredynamic where the infrared pattern changes temporarily, so that anyresidual pattern-to-depth dependence may be reduced.

3D Scanner with Statistical Pattern Generator

In another embodiment, the 3D scanner 160 includes at least one imagesensor 205(a-n), at least one lens, and a statistical pattern generator165. The statistical pattern generator may generator random patternsand/or pseudo-random patterns. The random patterns and/or pseudo-randompatterns may be projected to the patient to facilitate the 3D scanningof patient anatomy. The improved 3D scan may enhance the imageregistration between intraoperative 3D scan and preoperative images(e.g. MRI and CT), thereby improving the surgical navigation. Examplesof random patterns and/or pseudo-random patterns that the statisticalpattern generator 165 may generate may be shown in a statistical patternconfiguration 300 shown in FIG. 3. For instance, the random patterns310(a-n), where n is an integer equal to or greater than one, may bevery dense: for example, between 20,000 and 300,000 random dots may beused. Different from previous embodiments, only one image sensor205(a-n) is needed for this embodiment using a statistical patterngenerator 165.

The surgical imaging and navigation system 100 may include a 3D scanner160 that includes a projector, an image sensor 205 n, and a patterngenerator 165. The pattern generator 165 may generate a pseudo randompattern 310(a-n) that includes a plurality of dots. Each position ofeach corresponding dot included in the pseudo random pattern 310(a-n)may be pre-determined by the pattern generator 165. The projector mayproject the pseudo random pattern 310(a-n) onto the patient. Eachposition of each corresponding dot included in the pseudo random pattern310(a-n) is projected onto a corresponding position on the patient. Theimage sensor 205 n may capture a 2D intraoperative image of a pluralityof object points 250(a-n) associated with the patient.

The 3D scanning controller 150 may associate each object point 250(a-n)associated the patient that is captured by the image sensor 205 n with acorresponding dot included in the pseudo random pattern 310(a-n) that isprojected onto the patient by the projector based on the position ofeach corresponding dot as pre-determined by the pattern generator 165.The 3D scanning controller 150 may convert the 2D intraoperative image260 n to the 3D intraoperative scan of the patient based on theassociation of each object point 250(a-n) to each position of eachcorresponding dot included in the pseudo random pattern aspre-determined by the pattern generator 165.

The controller 110 may co-register pre-operative image data capturedfrom at least one pre-operative image of the patient with intraoperativeimage data provided by the 3D intraoperative image scan. The controller110 may instruct a display 120 to display the co-registeredpre-operative image data as captured from at least one pre-operativeimage with the intraoperative image data provided by the 3Dintraoperative scan as the surgeon navigates during the surgicalnavigation.

The projector may project the pseudo random pattern 310(a-n) onto a 2Dsurface before the patient is positioned on the 2D surface. The imagesensor 205 n may capture a 2D image of the pseudo random pattern ontothe 2D surface before the patient is positioned on the 2D surface. The3D scanning controller 150 may calibrate each position of each dotincluded in the pseudo random pattern 310(a-n) as projected onto the 2Dsurface and pre-determined by the pattern generator 165 to eachcorresponding position of each dot as included in the 2D image 260 ncaptured by the image sensor 205 n. The 3D scanning controller 150 maycompare each positon of each dot included in the pseudo random pattern310(a-n) as projected onto the 2D surface and pre-determined by thepattern generator 165 to each position of each dot included in thepseudo random pattern 310(a-n) as projected onto the patient. The 3Dscanning controller 150 may determine each depth of each object point250(a-n) as captured in the 2D intraoperative image 260 n by the imagesensor 205 n of the patient when the projector projects the pseudorandom pattern 310(a-n) onto the patient after the calibration based ona difference in depth of each corresponding dot included in the pseudorandom pattern 310(a-n) as projected onto the 2D surface as compared toeach corresponding dot included in the pseudo random pattern 310(a-n) asprojected onto the patient. The 3D scanning controller 150 may convertthe 2D intraoperative image 205 n to the 3D intraoperative scan of thepatient based on the depth of each object point 250(a-n) as provided bythe calibration of the pseudo random pattern 310(a-n) to the 2Dintraoperative image 260 n.

The 3D scanning controller 150 may determine a plurality of firstepipolar lines 220 a associated with a projection image plane 265 a ofthe projection of the pseudo random pattern 310(a-n) and a plurality ofsecond epipolar lines 220 n associated with the 2D intraoperative imageplane 265 n of the captured 2D intraoperative image 260 n based on anepipolar plane 210 that triangulates the plurality of object points250(a-n) included in the 2D intraoperative image 260 n to the pluralityof dots 250(a-n) included in the pseudo random pattern 310(a-n). Eachepipolar line 220(a-n) provides a depth of each object point 250(a-n) asprojected from the projection image plane 265 a associated with theprojector and the 2D intraoperative image plane 265 n associated withthe 2D intraoperative image 260 n. The 3D scanning controller 150 mayconvert the 2D intraoperative image 260 n to the 3D intraoperative imagescan of the patient based on the depth of each object point 250(a-n)provided by each corresponding epipolar line 220(a-n).

In one example, the statistical pattern generator 165 may include one ormore edge emitting laser, at least one collimating lens, and at leastone diffractive optics element. The edge emitting laser and thediffractive optics element may be controlled by the 3D scanningcontroller 150 to generate patterns 310(a-n) desirable for the 3Dscanning application in medicine. The edge emitting laser and thediffractive optics element generated patterns 310(a-n) that may beprojected on the patient anatomy, which increases the accuracy and speedof the 3D scanning. Improved 3D scan may enhance the image registrationbetween the intraoperative 3D scan and preoperative images (e.g. MRI andCT), thereby improving the surgical navigation. It should be appreciatedthat other illumination devices such as LEDs, a halogen lamp, a xenonlamp, an arc lamp, a laser diode may be used instead of an edge emittinglaser. In one example, the pattern 310(a-n) generated is dynamic wherethe pattern changes temporarily, so that any residual pattern-to-depthdependence may be reduced.

In another example, the statistical pattern generator 165 includes atleast one patterned vertical cavity semiconductor emission laser array,at least one collimating lens, and at least one diffractive opticselement. The patterned vertical cavity semiconductor emission laserarray and the diffractive optics element may be controlled by the 3Dscanning controller 150 to create patterns desirable for the 3D scanningapplication in medicine. The patterned vertical cavity semiconductoremission laser array and the diffractive optics element generatespatterns 310(a-n) that may be projected on the patient anatomy, whichincreases the accuracy and speed of the 3D scanning. The improved 3Dscan may enhance the image registration between intraoperative 3D scanand preoperative images (e.g. MRI and CT), thereby improving thesurgical navigation. In one example, the patterns 310(a-n) generated aredynamic where the pattern changes temporarily, so that any residualpattern-to-depth dependence may be reduced.

In another example, the statistical pattern generator 165 includes oneor more laser diodes and a statistically patterned aperture. Thestatistically patterned aperture may be made of metals, ceramics orplastics. In one aspect, the statistically patterned apertures with thelaser diodes created patterns 310(a-n) that may be combined with theinformation of the patient anatomy, which increases the accuracy andspeed of the 3D scanning. Improved 3D scan may enhance the imageregistration between intraoperative 3D scan and preoperative images(e.g. MRI and CT), thereby improving the surgical navigation. It shouldbe appreciated that other illumination devices such as halogen lamp,xenon lamp, arc lamp, LED may be used instead of a laser diode.

In another embodiment, the 3D scanner 160 includes oneinfrared-sensitive image sensor 205 n, an infrared-compatible lens, anoptical filter, and an infrared statistical pattern generator 165. Inone example, the optical filter is used in conjunction with theinfrared-sensitive image sensor 205 n and infrared-compatible lens tocapture infrared images (e.g. optical filter passes through at leastpart of the infrared spectrum). In one example, the infrared range isbeyond 800 nm. The optical filter may be a bandpass filter or along-pass filter (e.g. 800 nm long pass filters or 830 nm band passfilters). The infrared-sensitive image sensor 205 n may be a high-speedinfrared-sensitive CMOS image sensor. In one example, the infraredpattern generator 165 includes one or more laser diode and astatistically patterned aperture. In another example, the infraredstatistical pattern generator 165 includes one or more infrared edgeemitting laser, at least one infrared-compatible collimating lens, andat least one diffractive optics element. In yet another example, theinfrared statistical pattern generator 165 includes at least oneinfrared patterned vertical cavity semiconductor emission laser array,at least one infrared-compatible collimating lens, and at least onediffractive optics element. It should be appreciated that other infraredillumination devices such as an infrared halogen lamp, an infrared xenonlamp, an infrared arc lamp, an infrared laser diode may be used. In oneexample, the infrared pattern 310(a-n) generated may be dynamic wherethe infrared pattern 310(a-n) changes temporarily, so that any residualpattern-to-depth dependence may be reduced.

3D Scanner with Non-Statistical Projection Pattern Generator

In another embodiment, the 3D scanner 160 includes at least one imagesensor 205(a-n), at least one imaging lens, and a non-statisticalprojection pattern generator 165. The non-statistical projection patterngenerator 165 may create dynamic patterns 410(a-n), where n is aninteger equal to or greater than one, that are non-statistical. In anexample, the non-statistical projection pattern generator 165 maygenerate dynamic patterns 410(a-n) with spatial coding in spatialdomain, frequency domain, or a combination thereof. The patterns410(a-n) may be projected to the patient to facilitate the 3D scanningof patient anatomy. The patterns 410(a-n) may be projected dynamically:a series of patterns 410(a-n) are projected to properly encode thespatial information to facilitate 3D scanning, greatly reducingpattern-to-depth dependence. Improved 3D scan may enhance the imageregistration between intraoperative 3D scan and preoperative images(e.g. MRI and CT), thereby improving the surgical navigation.

The surgical imaging and navigation system 100 may include a 3D scanner160 that includes a projector, an image sensor 205 n, and a patterngenerator 165. The pattern generator 165 may generate a plurality ofnon-statistical patterns 410(a-n) with each non-statistical pattern410(a-n) including a plurality of identified characteristics. Eachplurality of identified characteristics associated with eachnon-statistical pattern 410(a-n) may have different variations of eachother. The projector may project each non-statistical pattern 410(a-n)onto the patient in series. Each variation in the identifiedcharacteristics of each non-statistical pattern 410(a-n_ as projectedonto the patient is adjusted based on when in the series eachcorresponding non-statistical pattern 410(a-n) is projected onto thepatient. The image sensor 205 n may capture a 2D intraoperative image260 n of a plurality of object points 250(a-n) with the patient aftereach non-statistical pattern 410(a-n) is projected onto the patient.

The 3D scanning controller 150 may identify a position of each objectpoint 250(a-n) associated with the patient that is captured by the imagesensor 205 n after each non-statistical pattern 410(a-n) is projectedonto the patient. The 3D scanning controller 150 may determine an actualposition of each object point 250(a-n) after the plurality ofnon-statistical patterns 410(a-n) is projected onto the patient based onan average position of each object point 250(a-n) determined from eachidentified position of each object point 250(a-n) as generated aftereach non-statistical pattern 410(a-n) is projected onto the patient. The3D scanning controller 150 may convert the 2D intraoperative image 260 nto the 3D intraoperative scan of the patient based on the actualposition of each object point 250(a-n) after the plurality ofnon-statistical patterns is projected onto the patient.

The controller 110 may co-register pre-operative image data capturedfrom at least one pre-operative image of the patient with intraoperativeimage data provided by the 3D intraoperative image scan. The controller110 may instruct the display 120 to display the co-registeredpre-operative image data as captured from the at least one pre-operativeimage with the intraoperative image data provided by the 3Dintraoperative scan as the surgeon navigates during the surgicaloperation.

The pattern generator 165 may generate the plurality of non-statisticalpatterns 410(a-n) with each non-statistical pattern 410(a-n) being avariation in scale from each other non-statistical pattern 410(a-n) thatis projected onto the patient. The pattern generator 165 may generate afirst non-statistical pattern that includes a strip with a resolutionthat is decreased to a resolution that the projector is capable toproject and the image sensor 205 n is capable to capture. The patterngenerator 165 may generate each additional non-statistical pattern410(a-n) that includes a stripe being an increased variation in scalefrom the first non-statistical pattern 410(a-n) and each additionalnon-statistical pattern 410(a-n) is a variation from each otheradditional non-statistical pattern 410(a-n) in the resolution of eachstripe associated with each additional non-statistical pattern 410(a-n).

The projector may project each non-statistical pattern 410(a-n) thatvaries in resolution to each corresponding horizontal row of pixelsincluded in the 2D intraoperative image 260 n captured by the imagesensor 205 n. The projector may project each non-statistical pattern410(a-n) that varies in resolution to each corresponding vertical columnof pixels included in the 2D intraoperative image 260 n captured by theimage sensor 205 n. The 3D scanning controller 150 may determine eachdepth of each object point 250(a-n) as captured in the 2D intraoperativeimage 260 n by the image sensor 205 n of the patient based on a depthassociated with each pixel included in the 2D intraoperative image 260 nthat is determined after each non-statistical pattern 410(a-n) isprojected onto the patient. The 3D scanning controller 150 may convertthe 2D intraoperative image 260 n to the 3D intraoperative scan of thepatient based on the depth of each object point 250(a-n) as determinedafter the plurality of statistical patterns 410(a-n) is projected ontothe patient.

The 3D scanning controller 150 may determine a plurality of firsteipolar lines 220 a with a projection image plane 265 a of theprojection of the plurality of non-statistical patterns 410(a-n) and aplurality of second epipolar lines 220 n associated with a 2Dintraoperative image plane 265 n of the captured 2D intraoperative image260 n based on an epipolar plane 210 that triangulates the plurality ofobject points 250(a-n) generated when each non-statistical pattern410(a-n) is applied to the 2D intraoperative image 260 n to theplurality of object points 250(a-n) included in the 2D intraoperativeimage 260 n. Each epipolar line 220(a-n) provides a depth of each objectpoint 250(a-n) as projected from the projection image plane 265 aassociated with the projector and the 2D intraoperative image plane 265n associated with the 2D intraoperative image 260 n. The 3D scanningcontroller 150 may convert the 2D intraoperative image 260 n to the 3Dintraoperative scan of the patient based on the depth of each objectpoint 250(a-n) provided by each corresponding epipolar line 220(a-n).

In one embodiment, the non-statistical projection pattern generator 165includes one or more LEDs, at least one lens, and a digital micromirrordevice. The LED and the digital micromirror device may be controlled bythe 3D scanning controller 150 to generate patterns 410(a-n) desirablefor the 3D scanning application in medicine. In another embodiment, thenon-statistical projection pattern generator 165 includes one or moreLEDs, at least one lens, and a thin-film-transistor liquid-crystaldisplay 120. The LED and thin-film-transistor liquid-crystal display 120may be controlled by the 3D scanning controller 150 to generate patterns165 desirable for the 3D scanning application in medicine. It should beappreciated that other illumination devices such as a halogen lamp, axenon lamp, an arc lamp, a laser diode may be used instead of an LED.

In yet another embodiment, the 3D scanner 160 includes at least oneinfrared-sensitive image sensor 205 n, at last one infrared-compatibleimaging lens, at least one optical filter, and an infrarednon-statistical projection pattern generator 165. In one example, theoptical filter is used in conjunction with the infrared-sensitive imagesensor 205 a and the infrared-compatible lens to capture infrared images205 n. In one example, the infrared range are beyond 800 nm. The opticalfilter may be a bandpass filter or a long-pass filter (e.g. 800 nm longpass filters or 830 nm band pass filters). The infrared-sensitive imagesensor 205 n may be a high-speed infrared-sensitive CMOS image sensor.In one example, the infrared non-statistical projection patterngenerator 165 includes one or more infrared LEDs, at least oneinfrared-compatible lens, and a digital micromirror device. The infraredLED and the digital micromirror device may be controlled by the 3Dscanning controller 150 to generate dynamic infrared patterns 410(a-n)desirable for the 3D scanning application in medicine. In anotherembodiment, the infrared non-statistical projection pattern 165 includesone or more infrared LEDs, at least one infrared-compatible lens, and athin-film-transistor liquid-crystal display 120. The infrared LED andthin-film-transistor liquid-crystal display 120 may be controlled by the3D scanning controller 150 to create dynamic infrared patterns 410(a-n)desirable for the 3D scanning application in medicine. It should beappreciated that other infrared illumination devices such as an infraredhalogen lamp, an infrared xenon lamp, an infrared arc lamp, an infraredlaser diode may be used instead of an infrared LED.

With the aforementioned apparatuses and systems, the dynamic projectionpattern 410(a-n) may be created to facilitate 3D scanning. A fewexamples of dynamic patterns 410(a-n) that the non-statisticalprojection pattern generator 165 creates are shown in non-statisticalpattern configuration 400 in FIG. 4 such as binary code, stripe boundarycode, and miere pattern. In one embodiment, binary codeword 410 a isrepresented by a series of black and white stripes. If black represents1 and white represents 0, the series of 0 and 1 at any given locationmay be encoded by the dynamic projection pattern 410 a (variestemporarily); the binary dynamic projection pattern 410 a may becaptured by the image sensor 205 n and lens, and decoded to recover thebinary codeword that encodes an location (e.g. 10100011). In theory, Nbinary patterns 410 a may generate 2N different codewords per imagedimension (x or y dimension). A representative binary pattern 410 isillustrated in FIG. 4. Similarly, binary coding may be extended toN-bits coding. For example, instead of binary case where only 1 and 0are represented by black and white, a N-bits integer may be representedby an intensity in between. For instance, if it is a 2-bit encodingsystem, 2²=4 different possibilities. If maximum intensity is I, 0, 1,2, 3 may be represented by I, ⅔*I, ⅓ *I, and 0, respectively. In otherexamples, dynamic stripe boundary code-based projection or the dynamicMoire code-based projection may be implemented.

In another embodiment, dynamic Fourier transform profilometry may beimplemented by the aforementioned apparatuses and systems. In oneaspect, periodical signals are generated to carry the frequency domaininformation including spatial frequency and phase. Inverse Fouriertransform of only the fundamental frequency results in a principle phasevalue ranging from −π to π. After spatial or temporal phase unwrapping(The process to remove 2π discontinuities and generate continuous map),actual 3D shape of patient anatomy may be recovered. Fourier transformprofilometry is less sensitive to the effect of out-of-focus images ofpatients, making it a suitable technology for intraoperative 3Dscanning. Similarly, π-shifted modified Fourier transform profilometrymay be implemented intraoperatively, where a π-shifted pattern is addedto enable the 3D scanning.

In another example, a DC image may be used with Fourier transformprofilometry. By capturing the DC component, the DC-modified Fouriertransform profilometry may improve 3D scan quality intraoperatively. Inanother example, N-step phase-shifting Fourier transform profilometrymay be implemented intraoperatively. It should be appreciated that thelarger the number of steps (N) is chosen, the higher the 3D scanningaccuracy. For instance, three-step phase-shifting Fourier transformprofilometry may be implemented to enable high speed 3D scanningintraoperatively. It should be appreciated that periodical patterns suchas trapezoidal, sinusoidal, or triangular pattern may be used in theFourier transform profilometry for intraoperative 3D scan. It should befurther appreciated that windowed Fourier transform profilometry,two-dimensional Fourier transform profilometry, or wavelet Fouriertransform profilometry may also be implemented by the aforementionedapparatuses and systems. It should be appreciated more than onefrequency of periodical signal (e.g. dual frequencies) may be used inthe modified Fourier transform profilometry, so that phase unwrappingbecome optional in the intraoperative 3D scan. The dynamic Fouriertransform profilometry and modified Fourier transform profilometrydiscussed herein may improve the quality of 3D scan of the patient.Improved 3D scan may enhance the image registration betweenintraoperative 3D scan and preoperative images (e.g. MRI and CT),thereby improving the surgical navigation.

In yet another embodiment, the aforementioned apparatuses and systemsimplement Fourier transform profilometry or modified Fourier transformprofilometry, in combination with binary codeword projection. TheFourier transform profilometry and binary codeword projection may beimplemented sequentially, concurrently, or a combination thereof. Thecombined approach may improve the 3D scanning accuracy, albert at thecost of 3D scanning speed. Improved 3D scan may enhance the imageregistration between intraoperative 3D scan and preoperative images(e.g. MRI and CT), thereby improving the surgical navigation.

In another embodiment, the aforementioned non-statistical projectionpattern generator may include at least one lens. The lens is configuredsuch a way so that the projected pattern(s) are defocused. The processof defocusing a binary pattern is illustrated in the defocusingconfiguration 500 depicted in FIG. 5. The defocusing process by the lensis similar a convolution of gaussian filter on the binary pattern.Consequently, the defocused binary pattern may create periodicalpatterns that are similar to sinusoidal patterns.

In another example, dithering techniques are used to generatedhigh-quality periodical fringe patterns through binarizing a higherorder bits fringe pattern (e.g. 8 bits) such as sinusoidal fringepatterns. In one example, ordered dithering is implemented; for example,Bayer matrix may be used to enable ordered dithering. In anotherexample, error-diffusion dithering is implemented; for instance,Floyd-Steinberg (FS) dithering or minimized average error dithering maybe implemented. It should be appreciated that in some cases thedithering techniques may be implemented in combination with defocusingtechnique to improve the quality of intraoperative 3D scan.

The 3D scanning controller 150 controls the modes and properties of 3Dscanner 160. For instance, the size of the area of 3D scanning, theresolution of 3D scanning, the speed of 3D scanning, the timing of 3Dscanning may be controlled by the 3D scanning controller 150. The 3Dscanning controller 150 may also implement the aforementioned methods of3D scanning. It should also be appreciated that the 3D scanningcontroller 150 may include the necessary hardware, software, orcombination thereof to carry out the 3D scanning methods previouslydiscussed. The 3D scanning controller 150 may include a microcontroller,a Field Programmable Gate Array (FPGA), a mobile or desktop computerthat may include a graphics processing unit (GPU), anapplication-specific integrated circuit (ASIC), or a combinationthereof.

Imaging Detector and Imaging Controller

The imaging detector 180 can capture the images intraoperatively. In oneembodiment, the imaging detector 180 is a color image sensor with alens. In one aspect, the color camera uses a Bayer filter pattern at thepixel level to detect color components of red, green and blue. Inanother example, the imaging detector 180 is a monochrome camera thatcan detect near infrared signal. It should be appreciated that in somecases the 3D scanner 160 already enable intraoperative imaging;therefore, no additional imaging detector is needed. In one example, the3D scanner 160 comprises at least one color image sensor and a lens, andthe color image camera and the lens can also serve as the imagingdetector 180 to capture intraoperative color image. Therefore, theimaging detector 180 and the 3D scanner 160 share hardware includingsaid color image sensor and said lens. In another example, the 3Dscanner 160 comprises at least one infrared-sensitive image sensor andan infrared-compatible lens, and the infrared-sensitive image sensor andan infrared-compatible lens can also serve as the imaging detector 180to capture intraoperative infrared image. Therefore, the imagingdetector 180 and the 3D scanner 160 share hardware including saidinfrared-sensitive image sensor and said infrared-compatible lens.

In another embodiment, the imaging detector 180 enable special purposeimaging, such as fluorescence imaging, hyperspectral imaging, thermalimaging, polarization imaging, photoacoustic imaging, etc. In oneexample, the imaging detector comprises a monochrome camera and afluorescence emission filter. For instance, the fluorescence filter maybe an 830 nm band pass filter to enable imaging of indocyanine green.

In another embodiment, the imaging detector 180 enable otherintraoperative imaging modalities. In one example, the imaging detector180 is an ultrasound transducer; therefore, intraoperative ultrasoundmay be enabled. In another example, the imaging detector 180 is afluoroscope; therefore, fluoroscopy may be enabled (2D or 3Dfluoroscopy). In yet another example, the imaging detector 180 is aC-arm X-ray scanner. In yet another example, the imaging detector 180 isan x-ray computed tomography scanner (CT); therefore, intraoperative CTmay be enabled. In yet another example, the imaging detector is aMagnetic resonance imaging (MRI) scanner; therefore, intraoperative MRImay be enabled; in yet another example, the imaging detector 180 is aoptical coherence tomography (OCT) scanner; therefore, intraoperativeOCT scanning may be enabled.

The imaging controller 170 can control the acquisition, storage, andprocessing of the images captured by the imaging detector 180. In oneaspect, the frame rate, field of view, magnification, and gain level ofimaging detector 180 may be controlled. In another aspect, the imagingcontroller 170 can synchronize the image acquisition with the modulationand frequency of the illumination to enable synchronized imagingcapturing under modulated illumination. In another example, the imagingcontroller 170 controls the acquisition and reconstruction ofintraoperative CT scan. In yet another example, the imaging controller170 controls the acquisition and reconstruction of intraoperative MRIscan. When there is more than one mode of imaging, the imagingcontroller 170 can also select and mode of image acquisition.

It should be appreciated that the imaging detector may share commonhardware with other components of the systems. In one example, the 3Dscanner 160 already enable color imaging; therefore, no standaloneimaging detector is needed. In another example, the tracker 130 alreadyenable infrared imaging; therefore, no standalone imaging detector isneeded.

It should also be appreciated that the imaging controller 170 mayinclude the necessary hardware, software, or a combination thereof tocarry out the imaging functions previously discussed. The imagingcontroller 170 may comprise a microcontroller, a Field Programmable GateArray (FPGA), a mobile or desktop computer that may include a graphicsprocessing unit (GPU), an application-specific integrated circuit(ASIC), or a combination thereof.

Light Source and Light Source Controller

The light source 195 can provide general-purpose and/or special-purposeillumination for surgical guidance. In one embodiment, the light source195 is a plurality of light emitting diodes (LEDs). The LEDs may bearranged in such a way to minimize the shadows produced by individualLEDs. A plurality of individual LEDs may be spaced apart to projectlight onto the patient, so that shadow cast by an intervening object isnegated by at least one other of a plurality of individual LEDs.

It should also be appreciated that the light source 195 may be based onother technologies, such as incandescent light lamp, laser diode,arc-lamp, laser, as well as coherent or in-coherent light sources. Itshould also be appreciated that the light source 195 may also includeone or a plurality of light diffusers to homogenize the illumination. Itshould also be appreciated that the light source 195 may also includeone or a plurality of collimation lenses to collimate the illumination.

In another embodiment, the light source 195 provides fluorescenceexcitation for fluorescence imaging, in conjunction with theaforementioned imaging detector 180. The light source 195 may compriseone or a plurality of spectral filters. In one aspect, one or more 775nm low pass filters may be used with white and near infrared LEDs toenable fluorescence excitation with indocyanine green.

In some embodiments, the light source controller 190 may control thelight source 195 to provide pulsed and/or modulated illumination.Frequency modulation, pulse-duration modulation, amplitude modulation,or phase modulation may be implemented. In one aspect, the illuminationis modulated at a frequency so that the illumination does not interferewith the 3D scanning performed by the 3D scanner 160. In another aspect,the illumination is modulated to be a DC signal so that the illuminationdoes not interfere with the 3D scanning performed by the 3D scanner 160.In yet another aspect, the illumination is modulated at a frequency sothat the illumination may be detected by the imaging detector 180 andimaging controller 170.

The light source controller 190 can control the intensity, mode,frequency, modulation of the light source 195. In one aspect, the lightsource controller 190 can synchronize the image acquisition with themodulation and frequency of the illumination to enable synchronizedimaging capturing under modulated illumination. When there is more thanone mode of illumination, the imaging controller 170 can also select themode of illumination provided by the light source 195. In anotheraspect, the light source 195 is synchronized with the 3D scanner 160 toenable 3D scanning and illumination in a sequentially interleavedfashion. In another aspect, the light source 195 is synchronized withthe tracker 130 to enable 3D scanning and tracking in a sequentiallyinterleaved fashion.

It should be appreciated that the light source 195 may share commonhardware with other components of the systems. In one example, the 3Dscanner 160 already enable surgical illumination; therefore, nostandalone light source is needed. In another example, the imagingdetector 180 already enable fluorescence excitation; therefore, nostandalone light source is needed.

Image Registration Process

The image registration between intraoperative 3D scan and other imagedata (e.g. preoperative/intraoperative CT or MRI) build correspondencesbetween patient anatomy and medical imageries. This can provide surgicalguidance and help surgical decision making of the surgeon.

The image registration is performed by the controller 110. In someembodiments, the controller 110 comprises a graphics processing unit(GPU) that can accelerate the image registration process. Pre-operativeor intraoperative image data such as CT (x-ray computerized tomography)or magnetic resonance imaging (MRI) may be registered to the 3D scanand/or surgical imaging provided by the system 100. The 3D image datacaptured by the system 100 may be in the form of point clouds, orpolygon mesh, or other formats that can represent 3D shape.

The user may define the regions of interest. For instance, L2 and L3 oflumber spine may be defined as the region of interest for imageregistration. The algorithm may automatically segment out certain organsor tissues to facilitate image registration (e.g. automatic segmentationand labelling of vertebrae based on the CT image). The definition ofregion-of-interest can expedite the registration process. Once the twodatasets (e.g. CT and intraoperative 3D scan) are registered, the fulldata set may be displayed (e.g. full lumber spine based on preoperativeCT data, instead of region of interest containing only L2 and L3).

In one embodiment, a surface-based image registration algorithm isimplemented for image registration. For surface-based imageregistration, the surface(s) of the intraoperative data and preoperativedata are matched. In one example, iterative closest point (ICP) methodis used for surface registration. For instance, the algorithm canminimize the difference between a first point cloud (representingintraoperative 3D data captured by the system 100) and a second pointcloud (representing preoperative 3D point cloud captured by the CT orMRI). In another example, a modified K-D tree algorithm may beimplemented with ICP for efficient closest point computation to enablesubset-subset matching. It should be appreciated that the ICP method maybe implemented with parallelization using GPU.

The 3D scanner 160 may acquire an intraoperative 3D scan of a surgicalfield. The medical image data (e.g., CT or MRI) may be loaded. Thecontroller 110 may perform segmentation on the medical image data (e.g.,CT or MRI) to isolate the organ of interest (e.g., 1.5 of lumber spinefor spine surgery). The controller 110 may reduce the image data intosurface data of the organ of interest (e.g., surface of L5 of lumberspine). The controller 110 may perform surface-based image registrationbetween the surface data of the organ of interest and the intraoperative3D scan.

It should be appreciated that in some embodiments only a subset of thesurface data of organ of interest is used. In one example, only theposterior portion of the surface data of vertebral body Lumber 5 (L5) isused for surface registration. It should also be appreciated that insome embodiments only a subset of the intraoperative 3D scan data isused. In one example, only the intraoperative 3D scan data near thesurgical field is used for surface registration.

The 3D scanner may acquire an intraoperative 3D scan of the surgicalfield. The controller 110 may load image data (e.g., CT or MRI). Thecontroller 110 may window/crop the image data to the neighborhood nearthe organ of interest (e.g., L5 of lumber spine for spine surgery). Thecontroller 110 may perform segmentation medical image data (e.g., CT orMRI) to isolate the organ of interest (e.g., L5 or lumber spine forspine surgery.) The controller 110 may reduce the image data intosurface data of organ of interest (e.g., surface of L5 of lumber spine).The controller 110 may perform surface-based image registration betweenthe surface data of organ of interest and the intraoperative 3D scan.

In one embodiment, the intraoperative 3D scan is spatially filtered ortrimmed. Therefore, only a subset of the intraoperative 3D scan is usedfor surface-based registration. The spatial filtering may be manual,automatic, or a combination thereof. In one example, the spatialfiltering is conducted per each vertebral level (L3, L4, L5). In anotherembodiment, the data density of the intraoperative 3D scan is adjusted.In one example, the point cloud representation of the intraoperative 3Dscan is down-sampled. In yet another embodiment, the intraoperative 3Dscan and medical image data are aligned with user input, prior to thesurface-based registration. The user identities and labelled a pluralityof common landmarks on the intraoperative 3D scan and the medical imagedata. The intraoperative 3D scan and the medical image data aresubsequently registered based on those landmarks. Thus, theintraoperative 3D scan and medical image data are aligned with userinput and landmark based registration, prior to the surface-basedregistration.

Different segmentation methods may be used. In one aspect,thresholding-based segmentation may be performed. For example, globalthresholding may be implemented for segmentation. In another example,adaptive thresholding may be implemented for segmentation. In anotherexample, segmentation may be performed based on statistical shape models(SSM). In another example, segmentation may be performed based onadaptive contouring. In yet another example, segmentation may beperformed based on machine learning such as artificial neural network,gradient boosting, or random forests. In another example, thesegmentation may be manual. Other segmentation methods that may beapplied are: Clustering methods. Motion & Interactive Segmentation,Compression-based methods, Histogram-based methods, Edge detection, Dualclustering method, Region-growing methods, Partial differentialequation-based methods, Variational methods, Graph partitioning methods(e.g Markov random fields (MRF), Supervised image segmentation usingMRF, Optimization algorithms, Iterated conditional modes/gradientdescent, Simulated annealing (SA), Unsupervised image segmentation usingMRF and expectation maximization, etc), Watershed transformation,Model-based segmentation, Multi-scale segmentation, One-dimensionalhierarchical signal segmentation, Image segmentation and primal sketch,Semi-automatic segmentation, Trainable segmentation, and combinationthereof.

In another embodiment, a feature-based image registration algorithm maybe implemented for image registration. A feature detection algorithm maybe used. In one example, scale-invariant feature transform (SIFT) isused for feature-based registration. In another example, speeded uprobust features (SURF) is used for feature-based registration. Inanother example, Gradient Location and Orientation Histogram is used forfeatured-based registration. In yet another example, histogram oforiented gradients (HOG) is used for featured-based registration. Itshould be appreciated that feature-based image registration algorithmmay be implemented on 3D point cloud or polygon meshes.

In one example, landmark based registration is implemented. The landmarkmay be anatomical or geometrical. For instance, a blood vessel or partof a bone may be used for landmark for registration. In another example,fluorescence tissues (e.g. tumors or blood vessels) may be used aslandmark based registration. In another example, segmentation-basedregistration is implemented. Rigid models (e.g. points, curves,surfaces, etc) or deformable models (e.g. snakes, nets, etc) may beimplemented.

In another example, fiducial based registration may be implemented. Forinstance, stereotactic frame, screw markers, mould, frame, dentaladapter, skin markers may be used as fiducials. In another example,machine learning algorithms are used for image registration. In oneaspect, supervised learning may be implemented. In another aspect,unsupervised learning may be implemented. In yet another aspect,reinforcement learning may be implemented. It should be appreciated thatfeature learning, sparse dictionary learning, anomaly detection,association rules may also be implemented. Various models may beimplemented for machine learning. In one aspect, artificial neuralnetworks are used. In another aspect, decision trees are used. In yetanother aspect, support vector machines are used. In yet another aspect,Bayesian networks are used. In yet another aspect, genetic algorithmsare used.

In yet another example, neural networks, convolutional neural networks,or deep learning are used for image segmentation, image registration, ora combination thereof. Neural network based systems are advantageous inmany cases for image segmentation, recognition and registration tasks. Aconvolutional neural network configuration 600 is depicted in FIG. 6A.In one example, Supervised Transformation Estimation is implemented; Inanother example, Unsupervised Transformation Estimation is implemented;In yet another example, Reinforcement Learning based Registration isimplemented; In yet another example, Deep Similarity based Registrationis implemented. In one example, U-net is implemented for imagesegmentation to isolate the organ or tissue of interest (e.g. vertebralbodies). An example of U-net architecture 620 is shown in FIG. 6A.

In one example, U-Net 620 has a contraction path and expansion path. Thecontraction path has consecutive convolutional layers and max-poolinglayer. The expansion path performs up-conversion and may haveconvolutional layers. The convolutional layer(s) prior to the outputmaps the feature vector to the required number of target classes in thefinal segmentation output. In one example, V-net is implemented forimage segmentation to isolate the organ or tissue of interest (e.g.vertebral bodies). In one example, Autoencoder based Deep LearningArchitecture is used for image segmentation to isolate the organ ortissue of interest. In one example, backpropagation is used for trainingthe neural networks.

In yet another example, deep residual learning is performed for imagerecognition or image segmentation, or image registration. A residuallearning framework is utilized to ease the training of networks. Aplurality of layers is implemented as learning residual functions withreference to the layer inputs, instead of learning unreferencedfunctions. One example of network that performs deep residual learningis deep Residual Network or ResNet.

In another embodiment, a Generative Adversarial Network (GAN) is usedfor image recognition or image segmentation, or image registration. Anexample of GAN configuration 650 is shown in FIG. 6B. In one example,the GAN 630 performs image segmentation to isolate the organ or tissueof interest. In the GAN 630, a generator 631 is implemented throughneural network to models a transform function which takes in a randomvariable 633 as input and follows the targeted distribution whentrained. A discriminator 632 is implemented through another neuralnetwork simultaneously to distinguish between generated data and truedata. In one example, the first network tries to maximize the finalclassification error between generated data and true data while thesecond network attempts to minimize the same error. Both networks mayimprove after iterations of the training process.

In yet another example, ensemble methods are used, wherein multiplelearning algorithms are used to obtain better predictive performance. Inone aspect, Bayes optimal classifier is used. In another aspect,bootstrap aggregating is used. In yet another aspect, boosting is used.In yet another aspect, Bayesian parameter averaging is used. In yetanother example, Bayesian model combination is used. In yet anotherexample, bucket of models is used. In yet another example, stacking isused. In yet another aspect, a random forests algorithm is used. In yetanother aspect, an gradient boosting algorithm is used.

The 3D scanning controller 150 may acquire the intraoperative 3D scan ofthe surgical field. The controller 110 may load image data (e.g., CT orMRI). The controller 110 may perform segmentation on the medical imagedata (e.g., CT or MRI) to isolate the organ of interest (e.g., L5 oflumber spine for spine surgery. The controller 110 may reduce the imagedata into surface data of the organ of interest (e.g., surface of L5 oflumber spine). The controller 110 may use surface data of the organ ofinterest to perform surface-based image registration with theintraoperative 3D scan. The image registration uses a machine learningalgorithm.

Tracker and Tracking Controller:

The tracker 130 can track the surgical tools. The tracking controller140 controls how the tracker 130 tracks the surgical tools. The trackingmay be enabled via optical tracking, or electromagnetic tracking, or acombination thereof. In one aspect, the tracker 130 is an opticaltracker. In another aspect, the tracker 130 is an electromagnetictracker.

In one embodiment, the optical tracking is implemented through aplurality of reflective markers. The reflective marker may be a sphere,plates or other structures that are highly reflective. In anotherembodiment, the optical tracking is implemented through a plurality oflight emitting diodes (LEDs). The LEDs may be in the near infraredspectrum to enable accurate tracking. In one aspect, active markers suchas LEDs may be attached to one end of surgical tools, to locate theirlocations. NDI Optotrak systems are examples of optical tracking systemsthat may be used for this embodiment.

In another embodiment, a modulated infrared optical tracking method maybe utilized by the system. As such, the wavelength of the opticalemitters for tracking purposes (such as LEDs) may be different from thewavelength used by the 3D scanner and the wavelengths used for theintraoperative imaging. Methods, such as spectral filtering may be usedto facilitate the separation of wavelengths between the optical emitterfrom the tracker 130 from other signals. In another example, frequencymodulation may also be used to separate the signal from the trackingoptical emitters from background signals. Specifically, frequencyfilters may be used to separate the tracking signals.

In another example, the tracker 130 comprises an inertial measurementunit (IMU). In one aspect, the IMU has a combination of accelerometersand gyroscopes, and optionally magnetometers. Therefore, gyroscopictracking may be performed. In one aspect, the IMU may be attached to thepatient or a surgical tool. In another embodiment, video tracking may beperformed based on computer vision. Various object tracking algorithmsmay be implemented. In one aspect, optical flow algorithm is used forvideo tracking. If electromagnetic tracking is used, the tracker 130 mayincorporate small coils or similar electromagnetic field sensors andmultiple position measurement devices. The electromagnetic field sensorsmay be attached to the surgical tools and the patient, to locate theirlocations, respectively.

In one example, the tracing controller 140 first registers thepreoperative image data (preoperative or intraoperative CT or MRI) withthe intraoperative image data (e.g. 3D scan of the anatomy obtained bythe system); the tracking controller 140 subsequently tracking theregistration optically using a reference frame with reflective markers.Because the reference frame has a fixed location with respect to thepatient, tracking the reference frame (e.g. a Mayfield clamp withmarkers) can enable accurate tracking of the registration. In addition,surgical tools with markers/fiducials can also be tracked by the tracker130. Therefore, the relationship between the surgical tool and theregistration can established via the reference frame. The trackingcontroller 140 controls how the tracker 130 tracks the surgical toolsand other objects. It should be appreciated that the tracker 130 mayshare common hardware with other components of the systems. In oneexample, the 3D scanner already enable optical tracking; therefore, nostandalone tracker 130 is needed. In another example, the imagingdetector 180 already enable optical tracking; therefore, no standalonetracker is needed.

Display

The display 120 may be a digital or analog display for display themedical information to the user. In one embodiment, the display 120 is aflat panel 2D monitor or TV. In another embodiment, the display 120 is aflat panel 3D monitor or 3D TV. The 3D monitor/TV may need to work withpassive polarizers or active shutter glasses. In one aspect, the 3Dmonitor/TV is glass-free. It should be appreciated that the display 120may be a touchscreen, or a projector. In one example, the display 120comprises a half transparent mirror that can reflect projection ofimages to the eyes of the user. The images being projected may be 3D,and the user may wear 3D glasses (e.g. polarizer; active shutter 3Dglasses) to visualize the 3D image data reflected by the halftransparent mirror. The half transparent mirror may be placed on top ofthe surgical field to allow the user to see through the half transparentmirror to visualize the surgical field.

In another embodiment, the display 120 is a near-eye display. It shouldbe appreciated that the near eye may be 3D. It should be furtherappreciated that the near-eye display 120 may comprise LCD (liquidcrystal) microdisplays, LED (light emitting diode) microdisplays,organic LED (OLED) microdisplays, liquid crystal on silicon (LCOS)microdisplays, retinal scanning displays, virtual retinal displays,optical see through displays, video see through displays, convertiblevideo-optical see through displays, wearable projection displays,projection display, and the like. It should be the appreciated that thedisplay 120 may be stereoscopic to enable displaying of 3D content. Inanother embodiment, the display 120 is a projection display.

In one aspect, the display 120 is a digital 3D magnification devicecapable of enable different magnification levels at different levels. Inanother aspect, the display 120 is an augmented reality (AR) displaythat can display the surgical navigation and imaging data as part of theAR content. The display 120 can display the 3D scanning data,registration data, navigation data, original preoperative image data,intraoperative image data, or a combination thereof, to the user. In oneaspect, the display 120 can display the registration process, positionsof surgical tools, and tracking of registration. In another aspect, thedisplay may display intraoperative imaging data such as color imaging orfluorescence imaging data.

Controller

The controller 110 comprises the hardware and software necessary toimplement the aforementioned methods. In one embodiment, the controller110 involves a computer-readable medium comprising processor-executableinstructions configured to implement one or more of the techniquespresented herein. An example embodiment of a computer-readable medium ora computer-readable device comprises a computer-readable medium, such asa SSD, CD-R, DVD-R, flash drive, a platter of a hard disk drive, etc.,on which is encoded computer-readable data. This computer-readable data,such as binary data comprising at least one of a zero or a one, in turncomprises a set of computer instructions configured to operate accordingto one or more of the principles set forth herein. In some embodiments,the set of computer instructions are configured to perform a method,such as at least some of the exemplary methods described herein, forexample. In some embodiments, the set of computer instructions areconfigured to implement a system, such as at least some of the exemplarysystems described herein, for example. Many such computer-readable mediaare devised by those of ordinary skill in the art that are configured tooperate in accordance with the techniques presented herein.

The following discussion provide a brief, general description of asuitable computing environment to implement embodiments of one or moreof the provisions set forth herein. Example computing devices include,but are not limited to, personal computers that may comprise a graphicsprocessing unit (GPU), server computers, hand-held or laptop devices,mobile devices (such as mobile phones, Personal Digital Assistants(PDAs), media players, and the like), multiprocessor systems, consumerelectronics, mini computers, mainframe computers, a microcontroller, aField Programmable Gate Array (FPGA), an application-specific integratedcircuit (ASIC), distributed computing environments that include any ofthe above systems or devices, and the like.

Although not required, embodiments are described in the general contextof “computer readable instructions” being executed by one or morecomputing devices. Computer readable instructions may be distributed viacomputer readable media. Computer readable instructions may beimplemented as program components, such as functions, objects,Application Programming Interfaces (APIs), data structures, and thelike, that perform particular tasks or implement particular abstractdata types. Typically, the functionality of the computer readableinstructions may be combined or distributed as desired in variousenvironments.

In one example, a system comprises a computing device configured toimplement one or more embodiments provided herein. In one configuration,the computing device includes at least one processing unit and onememory unit. Depending on the exact configuration and type of computingdevice, the memory unit may be volatile (such as RAM, for example),non-volatile (such as ROM, flash memory, etc., for example) or somecombination of the two. In other embodiments, the computing device mayinclude additional features and/or functionality. For example, thecomputing device may also include additional storage (e.g., removableand/or non-removable) including, but not limited to, cloud storage,magnetic storage, optical storage, and the like. In one embodiment,computer readable instructions to implement one or more embodimentsprovided herein may be in the storage. The storage may also store othercomputer readable instructions to implement an operating system, anapplication program, and the like. Computer readable instructions may beloaded in the memory for execution by the processing unit, for example.

The term “computer readable media” as used herein includes computerstorage media. Computer storage media includes volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer readableinstructions or other data. Computer storage media includes, but is notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, Digital Versatile Disks (DVDs) or other optical storage,magnetic disk storage or other magnetic storage devices, or any othermedium which may be used to store the desired information and which maybe accessed by computing device.

The computing device may also include communication connection(s) thatallows the computing device to communicate with other devices.Communication connection(s) may include, but is not limited to, a modem,a Network Interface Card (NIC), an integrated network interface, a radiofrequency transmitter/receiver, an infrared port, a USB connection, orother interfaces for connecting computing device to other computingdevices. Communication connection(s) may include a wired connection or awireless connection. Communication connection(s) may transmit and/orreceive communication media.

The computing device may include input device(s) such as keyboard,mouse, pen, voice input device, touch input device, infrared cameras,depth cameras, touchscreens, video input devices, and/or any other inputdevice. Output device(s) such as one or more displays, speakers,printers, and/or any other output device may also be included in thecomputing device. Input device(s) and output device(s) may be connectedto the computing device via a wired connection, wireless connection, orany combination thereof. In one embodiment, an input device or an outputdevice from another computing device may be used as input device(s) oroutput device(s) for computing device.

Components of computing device 6712 may be connected by variousinterconnects, such as a bus. Such interconnects may include aPeripheral Component Interconnect (PCI), such as PCI Express, aUniversal Serial Bus (USB), firewire (IEEE 1394), an optical busstructure, and the like. In another embodiment, components of computingdevice may be interconnected by a network. For example, the memory maybe comprised of multiple physical memory units located in differentphysical locations interconnected by a network.

Those skilled in the art will realize that storage devices utilized tostore computer readable instructions may be distributed across anetwork. For example, a computing device accessible via a network maystore computer readable instructions to implement one or moreembodiments provided herein. Computing device may access anothercomputing device and download a part or all of the computer readableinstructions for execution. Alternatively, the first computing devicemay download pieces of the computer readable instructions, as needed, orsome instructions may be executed at the first computing device and someat the second computing device.

Various operations of embodiments are provided herein. In oneembodiment, one or more of the operations described may constitutecomputer readable instructions stored on one or more computer readablemedia, which if executed by a computing device, will cause the computingdevice to perform the operations described. The order in which some orall of the operations are described should not be construed as to implythat these operations are necessarily order dependent. Alternativeordering will be appreciated by one skilled in the art having thebenefit of this description. Further, it will be understood that not alloperations are necessarily present in each embodiment provided herein.Also, it will be understood that not all operations are necessary insome embodiments.

It should be appreciated that the controller 110, the trackingcontroller 140, the light source controller 190, the intraoperativeimaging controller 170, and the 3D scanning controller 150 may sharecommon hardware and software with the other components of the systems.In one example, the controller 110 has necessary hardware to run thesoftware to control the 3D scanner 160; therefore, no standalone 3Dscanning controller 150 is necessary. In another example, the controllerhas necessary hardware to run the software to control the imagingdetector 180; therefore, no standalone intraoperative imaging controller170 is necessary. In yet another example, the controller 110 hasnecessary hardware to run the software to control the light source 195;therefore, no standalone light source controller 190 is necessary. Inyet another example, the controller has necessary hardware to run thesoftware to control the tracker 130; therefore, no standalone trackingcontroller 140 is necessary.

Systems with Master-Slave Dual 3D Scanners

In one embodiment, the system 700 comprises more than one 3D scanners,as shown in FIG. 7. In this invention, we denote the first 3D scanner asthe master 3D scanner 760 a, and the second 3D scanner as the slave 3Dscanner 760 n. In one aspect, the system further comprises a first 3Dscanning controller, denoted as the master 3D scanning controller 750 a.The master 3D scanning controller 750 a controls the master 3D scanner760 a to perform the 3D scan on organs and patients. In another aspect,the system 700 further comprises a second 3D scanning controller,denoted as the slave 3D scanning controller 750 n. The slave 3D scanningcontroller 750 n controls the slave 3D scanner 760 n to perform the 3Dscan on organs and patients. It should be appreciated that the master 3Dscanning controller 750 a and slave 3D scanning controller 750 n may beembodied as one single circuit, one single microcontroller, or onesingle computer that controls both master 3D scanner 760 a and slave 3Dscanner 760 n to perform 3D scans.

In one aspect, the system further comprises an intraoperative imagingcontroller 170 and an imaging detector 180. In one embodiment, themaster 3D scanner 760 a comprises a pattern creator. In one example, thepattern creator comprises at least one digital micromirror device (DMD),and said digital micromirror device are controlled by the 3D scanningcontroller to create patterns for 3D scanning. An instance of such apattern creator is Texas Instruments DLP products. In another example,the pattern creator comprises at least one light emitting diode, atleast one lens, and at least one thin-film-transistor liquid-crystaldisplay, and said light emitting diode and said thin-film-transistorliquid-crystal display are controlled by the master 3D scanningcontroller 750 a to create patterns for intraoperative 3D scanning.

In another embodiment, the master 3D scanner 760 a comprises astatistical pattern creator, wherein the statistical pattern creatorcreates random patterns or pseudo-random patterns and said patterns areprojected to the patient to facilitate the 3D scanning of patientanatomy. In one aspect, the statistical pattern creator may createdynamic statistical patterns that changes temporarily. In anotherembodiment, the master 3D scanner 760 a comprises a non-statisticalpattern creator, wherein the non-statistical pattern creator createsnon-statistical patterns are projected to the patient to facilitate the3D scanning of patient anatomy. In one aspect, the non-statisticalpattern is binary code, stripe boundary code, or sinusoidal code. Itshould be appreciated that all the non-statistical pattern describedpreviously may be applied here.

In one aspect, the slave 3D scanner 760 n uses similar components as themaster 3D scanner 760 a. For instance, both master 3D scanner 760 a andslave 3D scanner 760 n may use DMDs. In some cases, the master 3Dscanner 760 a and the slave 3D scanner 760 n are the same, hardwarewise. In one example, the master 3D scanner 760 a and the slave 3Dscanner 760 n share a single pattern creator that creates dynamicprojection patterns. The pattern creator may be a statistical patterncreator or a non-statistical pattern creator. The camera in the master3D scanner 760 a and the camera in the slave 3D scanner 760 n may besimilar or the same. Thus, the master 3D scanner 760 a and the slave 3Dscanner 760 n together has only one pattern creator but two cameras. Inone example, the pattern creator is a projector.

In another aspect, the slave 3D scanner 760 n uses different componentsfrom components of the master 3D scanner 760 a. In one example, themaster 3D scanner 760 a uses DMDs, but the slave 3D scanner 760 n has astatistical pattern creator that creates random patterns orpseudo-random patterns. In another example, the master 3D scanner 760 auses liquid-crystal display, but the slave 3D scanner 760 n has astatistical pattern creator that creates random patterns orpseudo-random patterns. In one aspect, the master 3D scanner 760 aperforms 3D scan of patients with higher resolution and slower speed,and the slave 3D scanner 760 n performs 3D scan of patients with lowerresolution and faster speed. This is advantageous in many cases. Forexample, the master 3D scanner 760 a can create a high resolution 3Dscan at the beginning of surgery (master 3D scan), and be idleafterwards; the slave 3D scanner 760 n can continuously scan patientwith a lower resolution 3D scan at high speed (slave 3D scan(s)). Thesystem can use slave 3D scan(s) to monitor the patient and registrationstatus. For example, if significant changes in surgical landscape isdetected automatically, the system can use the master 3D scanner 760 ato generate an updated master 3D scan for image registration andnavigation. In another example, the user can control the system to usethe master 3D scanner 760 a to generate master 3D scan on demand. Inanother example, the master 3D scanner 760 a captures a 3D scan with abigger field of view (FOV), and the slave 3D scanner 760 n captures a 3Dscan with a smaller field of view. Both the master 3D scan (bigger FOV)and slave 3D scan (smaller FOV) may be used for image registration.

The master 3D scanner 760 a and slave 3D scanner 760 n can work togetherin several different ways. In one aspect, the master 3D scanner 760 aand slave 3D scanner 760 n can perform 3D scanning concurrently,controlled by the master 3D scanning controller 750 a and the slave 3Dscanning controller 750 n, respectively. In another aspect, the master3D scanner 760 a and slave 3D scanner 760 n can perform 3D scanningsequentially, controlled by the master 3D scanning controller 750 a andthe slave 3D scanning controller 750 n, respectively. The sequentialmaster 3D scan and slave 3D scan can minimize the crosstalk betweenmaster 3D scanner 760 a and slave 3D scanner 760 n. For instance, thesystem performs master 3D scan first using the master 3D scanner 760 a,and subsequently performs slave 3D scan using the slave 3D scanner 760n.

In another aspect, the master 3D scanner 760 a and slave 3D scanner 760n can perform 3D scanning in an interleaved fashion, controlled by themaster 3D scanning controller 750 a and slave 3D scanning controller 750n, respectively. For instance, the system performs a first half ofmaster 3D scan; the system secondly performs a first half of slave 3Dscan; thirdly, the system performs the second half of master 3D scan,and the master 3D scan is completed; lastly, the system performs thesecond half of slave 3D scan, and the slave 3D scan is completed. Itshould be appreciated there are many ways to interleave the master 3Dscan and slave 3D scan; in one aspect, the master 3D scan is dividedinto a plurality of master 3D scan portions, and the slave 3D scan isdivided into a plurality of slave 3D scan portions. The master 3D scanportions and slave 3D scan portions are acquired in an interleavedfashion. An example of temporal sequence is: master 3D scan portion 1,slave 3D scan portion 1, master 3D scan portion 2, slave 3D scan portion2, master 3D scan portion 3, slave 3D scan portion 3, etc.

The master 3D scanner 760 a may capture a master 3D scan of an anatomyof a patient. The slave 3D scanner 760 n may capture a slave 3D scan ofan anatomy of a patient. The controller 110 may register the slave 3Dscan to the master 3D scan to generate a co-registered intraoperative 3Dscan. The controller 110 may load medical image data (e.g., CT or MRI).The controller 110 may perform segmentation on the medical image data(e.g., CT or MRI) to isolate the organ of interest (e.g., L5 of lumberspine for spine surgery). The controller 110 may reduce the image datainto the surface data of the organ of interest (e.g., surface of L5 oflumber spine). The controller 110 may perform surface-based imageregistration between the surface data of the organ of interest and theintraoperative 3D scan. The display 120 may display the results of thesurface-based image registration to the user.

In one embodiment, the master 3D scanner 760 a is a part of a surgicalnavigation system, and the slave 3D scanner 760 n is a part of asurgical tool, such as surgical drill. In another embodiment, there area plurality of slave 3D scanners 760 n. For instance, more than onesurgical tools each with a slave 3D scanner 760 n can work together. Inanother embodiment, there are a plurality of master 3D scanners 760 a.In one aspect, the master 3D scanner 760 a is positioned further awayfrom the patient, and the slave 3D scanner 760 n is positioned closer tothe patient. In another aspect, the master 3D scanner 760 a ispositioned at a first angle relative to the patient, and the slave 3Dscanner 760 n is positioned at a second angle relative to the patient.Different angles relative to patient and different distances frompatient can help the system to capture 3D scans without blind spots dueto obstruction of line of sight by obstacles such as surgicalinstruments and surgeon's arms.

In another embodiment, the system further comprises a tracker and atracking controller, in addition to the other components shown in FIG.7. The image registration may be passed to the tracking controller andtracker, and the tracker can track at least one object such as asurgical tool. In one example, the tracker uses optical tracking, suchas passive infrared tracking based on reflective spheres attached to thesurgical tools. In another example, the tracker uses electromagnetictracking. The master 3D scanner 760 a may capture a master 3D scan of ananatomy of a patient. The Slave 3D scanner 760 n may scan an anatomy ofa patient. The controller 110 may register the slave 3D scan to themaster 3D scan to generate a co-registered intraoperative 3D scan. Thecontroller 110 may load medical image data (e.g., CT or MRI). Thecontroller 110 may perform segmentation of medical image data (e.g, CTor MRI) to isolate the organ of interest (e.g., L5 of lumber spine forspine surgery). The controller 110 may reduce the image data into thesurface data of the organ of interest (e.g., surface of L5 of lumberspine). The controller 110 may perform surface-based image registrationbetween the surface data of the organ of interest and the intraoperative3D scan. The tracker may track positions of at least one entity in asurgery.

Smart Surgical Instruments with Navigation Capability

In one embodiment, the system 800 comprises a 3D scanning controller150, a 3D scanner 160, and a surgical tool 880, as shown in FIG. 8. The3D scanning controller 150 controls the 3D scanner 160 to capture anintraoperative 3D scan, and the intraoperative 3D scan may be used forimage registration and navigation. The image navigation can guide theplacement of the surgical tools such as a drill or saw. In one aspect,the system 800 further comprises a controller 110 and a display 120. Thecontroller 110 can perform the image registration process using an imageregistration algorithm, and the display 120 can display the imageregistration and navigation data to the user. In one aspect, the display120 is an LCD display or an OLED display attached to the surgical tool.

In one embodiment, the 3D scanner 160 is situated at the end of thesmart surgical tool 880 closer the patient, and the display 120 issituated at the end of the smart surgical tool 880 closer theuser/surgeon as shown in the surgical tool configuration 900 depicted inFIG. 9. Therefore, the 3D scanner 160 can capture a 3D scan of thepatient 888 without obstruction of the surgical tool 880, and thedisplay 120 can display the surgical navigation and registration data tothe user easily. In one aspect, the smart surgical tool 880 includes ahandle 890 so that it may be handheld by the user. In another aspect,the smart surgical tool 880 is mounted on a mechanical arm that may bepositioned manually or robotically. It should be appreciated that thesmart surgical tool 880 may be very small and light weight.

In one embodiment, the smart surgical tool 880 comprises a surgicalinstrument. Here are some examples of instruments that may be integratedas part of the smart surgical tool 880: graspers, forceps, clamps,occluders, needle drivers, retractors, distractors, positioners,stereotactic devices, mechanical cutters, scalpels, lancets, drill bits,rasps, trocars, ligasure, harmonic scalpel, surgical scissors, rongeurs,dilators, specula, suction tips, tubes, sealing devices, surgicalstaplers, irrigation and injection needles, tips and tubes, powereddevices, drills, saws, dermatomes, scopes, probes, endoscopes, tactileprobes, ultrasound tissue disruptors, cryotomes, cutting laser guides,measurement devices, etc.

In one embodiment, the instrument/tool in the smart surgical tool 880may be replaced with other compatible surgical tools and instruments.For example, a lancet initially installed in the smart surgical tool 880may be replaced with a trocar. The smart surgical tool 880 may be used avariety of surgical instruments to guide surgery. In one aspect, thesmart surgical tool 880 is an attachment to any compatible surgicalinstrument. In one example, the system comprises an attachment mechanismfor attaching surgical instruments and tools. Different surgical toolsand instruments may be attached or mounted to the system using theattachment mechanism. The attachment mechanism may be mechanical,chemical, electrical, or electromagnetic. The instruments may be mountedon, installed on, screwed into, clipped to, coupled to, slide into, orpushed into the system.

In another embodiment, the smart surgical tool 880 further comprises animaging detector and an intraoperative imaging controller. In oneexample, the imaging detector is a fluorescence imaging camera. Inanother example, the imaging detector is a color camera. Therefore,intraoperative imaging may be conducted by the smart surgical tool 880.In another example, the smart surgical tool 880 further comprises asurgical tool controller 110 that controls the surgical tool 880. In oneembodiment, the imaging detector is situated at the end of the smartsurgical tool 880 closer the patient 888, and the display 120 issituated at the end of the smart surgical tool 880 closer theuser/surgeon. Therefore, the imaging detector can capture anintraoperative image of the patient 888 without obstruction of thesurgical tool 880, and the display 120 can display the surgical imagingdata to the user easily. In another embodiment, the smart surgical tool880 comprises a robotic arm. The system 800 may be placed by the roboticarm under user's direction. The surgical tool 880 can also be trigger bythe user manually or automatically (e.g. start drilling and stopdrilling). The image registration and navigation data may be used toguide the placement and control of the tool and robotics.

Registration Per Individual Organ Level

In one embodiment, the registration may be performed at an individualorgan level. For example, the spine navigation, registration may begenerated per individual vertebrae level. For instance, for L3, L4, L5lumber fusion procedure, the registration may be performed based on L3vertebrae, based on L4 vertebrae, or based on L5 vertebrae,respectively. In one aspect, the system 1000 can generate differentimage masks (e.g. L3 mask, L4 mask, L5 mask) to spatially filter theintraoperative 3D scan data and/or preoperative CT data, forregistration at different levels. For example, the system 1000 canregister the intraoperative 3D scan only to the L5 CT data.

The 3D scanner 160 may acquire an intraoperative 3D scan of the surgicalfield. The controller 1010 may load medical image data (e.g., CT orMRI). The controller 1010 may perform segmentation of medical image data(e.g., CT or MRI) to isolate the organ of interest (e.g., L5 of lumberspine for spine surgery) and generate image mask for different organs(e.g. L3 mask, L4 mask, L5 mask). The controller 1010 may reduce theimage data into surface data of the organ of interest (e.g., surface ofL5 of lumber spine). The controller 1010 may use surface data of theorgan of interest (e.g., surface of L5 of lumber spine) to performsurface-based image registration with intraoperative 3D scan.

After image registration based on L5 vertebra is obtained, imageregistration based on L4 vertebra and L3 vertebra can also be generated.All of image registrations (e.g. L3 based registration, L4 basedregistration, and L5 based registration) or a subset of registration (L5based registration only) may be presented to the user. In anotherexample, the image registration may be performed at a level includingmore than one organ. For example, for navigation of L3, L4, L5 fusionsurgery, L3 and L4 may be used for image registration and navigation.The system 1000 can generate and an image mask including only L3 and L4vertebrae.

Monitoring Movements of Organs

In surgical navigation, organs often move as the surgery progresses. Forinstance, in spine surgeries, vertebrae often move after the initialimage registration is completed. The organ movement compromises theaccuracy of surgical navigation. Thus, it is advantageous to monitortissue movements after initial image registration. The 3D scanner 160may acquire a first operative 3D scan of the surgical field (initial 3Dscan). The 3D scanner 160 may acquire a second intraoperative 3D scan ofthe surgical field (subsequent 3D scan). The controller 1010 may comparethe first intraoperative 3D scan and the second intraoperative 3D scanto calculate the intraoperative 3D scan difference. The controller 1010may repeat organ movement when the intraoperative 3D scan difference isover the threshold.

It should be appreciated that the threshold of the intraoperative 3Dscan difference may be manually set, automatically set, proportionallyset (by percentage, e.g. 5%), or set using machine learning algorithms.The intraoperative 3D scans may be represented in different datastructures, such as point clouds, polygon meshes, or other datastructures. The controller 1010 may generate an image mask to includethe organs of interest. The 3D scanner 160 may use the image mask toacquire a first intraoperative 3D scan of the surgical field (initial 3Dscan). The 3D scanner 160 may use the image mask to acquire a secondintraoperative 3D scan of the surgical field (subsequent 3D scan). Thecontroller 1010 may compare the first intraoperative 3D scan and thesecond intraoperative 3D scan to calculate the intraoperative 3D scandifference. The controller 1010 may repeat organ movement when theintraoperative 3D scan difference is over the threshold.

In another example, the comparison between initial intraoperative 3Dscan and subsequent intraoperative 3D scan is performed based on imageregistration between the initial intraoperative 3D scan and thesubsequent intraoperative 3D scan. When there is little organ movement,the registration between initial intraoperative 3D scan and subsequentintraoperative 3D scan is good; when there is significant organmovement, the registration between initial intraoperative 3D scan andsubsequent intraoperative 3D scan is worse and there is a biggermisalignment. Therefore, the system can monitor the organ movement bymonitor the image registration between the initial intraoperative 3Dscan and subsequent intraoperative 3D scan. In one aspect, the imageregistration may be conducted using surface-based image registration. Inone example, the surface-based image registration may be performed usingiterative closest point (ICP) method. In one aspect, the system canreturn a confidence level score that indicates the wellness of theregistration. When the confidence score is high, less organ movement isreported; When the confidence score is low, more organ movement isreported.

The controller 1010 may generate an image mask to include the organs ofinterest. The 3D scanner 160 may use the image mask to acquire a firstintraoperative 3D scan of the surgical field (initial 3D scan). The 3Dscanner 160 may use the image mask to acquire a second intraoperative 3Dscan of the surgical field (subsequent 3D scan). The controller 1010 mayregister the first intraoperative 3D scan to the second intraoperative3D scan to generate an intraoperative registration confidence levelusing surface-based registration. The controller 1010 may report organmovement if the intraoperative registration confidence level is under athreshold.

It should be appreciated that the threshold of the intraoperativeregistration confidence level may be manually set, automatically set,proportionally set (by percentage, e.g. 5%), or set using machinelearning algorithms. The controller 1010 may generate an image mask toinclude the organs of interest. The 3D scanner 160 may use the imagemask to acquire a first intraoperative 3D scan of the surgical field(initial 3D scan). The controller 1010 may load medical image data(e.g., CT or MRI). The controller 1010 may perform segmentation medicalimage data (e.g., CT or MRI) to isolate the organ of interest (e.g., L5or lumber spine for spine surger) and generate image mask for organs ofinterest (e.g., L5 mask). The controller 1010 may use image data oforgans of interest to perform image registration between the initialintraoperative 3D scan and the image data. The controller 1010 may usethe image mask to acquire a second intraoperative 3D scan of thesurgical field (subsequent case). The controller 1010 may compare thefirst intraoperative 3D scan and the second intraoperative 3D scan tocalculate the intraoperative 3D scan difference. The controller 1010 mayreport organ movement if the intraoperative 3D scan difference is overthe threshold. The controller 1010 may use the image data of the organsof interest to perform image registration between subsequentintraoperative 3D scan and image data when the intraoperative 3D scandifference is over the threshold. The image registration may besurface-based, fiducial-based, landmark-based, featured-based or acombination thereof.

In another example, the image registration between CT or MRI data andintraoperative surgical field is conducted using fiducials (e.g.trackers) and intraoperative CT scans or 3D fluoroscopy. However, theorgan movement is monitored using the 3D scanner 160. When there is thesignificant organ movement, the system 1000 notifies the user to conductanother intraoperative CT scans or 3D fluoroscopy, to re-calculate theimage registration between CT or MRI data and intraoperative surgicalfield. The image registration may be surface-based, fiducial-based,landmark-based, featured-based or a combination thereof.

The controller 1010 may perform the intraoperative CT scans, 3Dfluoroscopy, or MRIs of the patient with fiducials to calculate theimage registration between image data (e.g., CT or MRI data) andintraoperative surgical field (initial fiducial-based registration). Thecontroller 1010 may generate an image mask to include the organs ofinterest. The controller 1010 may use the image mask to acquire a firstintraoperative 3D scan of the surgical field (initial 3D scan). Thecontroller 1010 may use the image mask to acquire a secondintraoperative 3D scan of the surgical field (subsequent 3D scan). Thecontroller 1010 may compare the first intraoperative 3D scan and thesecond intraoperative 3D scan to calculate the intraoperative 3D scandifference. The controller 1010 may report organ movement when theintraoperative 3D scan difference is over the threshold. The controller1010 may perform another intraoperative CT scan, 3D fluoroscopy or MRIto re-calculate the image registration between image data (e.g., CT orMRI data) and intraoperative surgical field (updated fiducial-basedregistration) when the intraoperative 3D scan difference is over thethreshold.

The controller 1010 may perform intraoperative CT scans, fluoroscopy, orMRI of patient with fiducials to calculate the image registrationbetween image data (e.g., CT or MRI data) and intraoperative surgicalfield (initial fiducial-based registration). The tracker 130 may trackthe image registration between image data and intraoperative surgicalfield using optical tracking or magnetic tracking (track initialfiducial-based registration). The controller 1010 may generate an imagemask to include the organs of interest. The 3D scanner 160 may use theimage mask to acquire a first intraoperative 3D scan of the surgicalfield (initial 3D scan). The 3D scanner 160 may use the image mask toacquire a second intraoperative 3D scan of the surgical field(subsequent 3D scan). The controller 1010 may compare the firstintraoperative 3D scan and the second intraoperative 3D scan tocalculate the intraoperative 3D scan difference. The controller 1010 mayreport organ movement when the intraoperative 3D scan difference is overthe threshold. The controller 1010 may perform another intraoperative CTscan, 3D fluoroscopy, or MRI, to re-calculate the image registrationbetween the image data (e.g., CT or MRI data) and intraoperativesurgical field (updated fiducial-based registration) when theintraoperative 3D scan difference is over the threshold. The tracker 130may track the image registration between image data and intraoperativesurgical field using optical tracking or magnetic tracking (trackupdated fiducial-base registration).

The aforementioned methods of tracking organ movements may be applied tovarious surgical subspecialties, such as orthopedic surgeries,neurosurgeries, spine surgeries, brain surgeries, cranial-facialsurgeries, cancer surgeries, plastic surgeries, general surgeries, etc.The aforementioned methods of tracking organ movements may be performedby an apparatus that includes a 3D scanner 160, a 3D scanning controller150, and a controller 1010 that calculates intraoperative 3D scandifference. The 3D scanning controller 150 instruct the 3D scanner 160to perform the initial intraoperative 3D scan and subsequentintraoperative 3D scan. The controller 1010 may be a computer, an ASIC,a digital circuit, an FPGA, or a combination thereof, running thealgorithm to calculate intraoperative 3D scan differences.

In another embodiment, the apparatus further comprises imaging detector180 for intraoperative imaging, tracker for tracking fiducials 130, andlight source 195 for illumination. In yet another embodiment, theapparatus 1000 comprises a communication interface 1080 with communicatewith other computers or surgical navigation systems as depicted in FIG.10. For example, when significant organ movement is detected, theapparatus 1000 notifies the other computer or surgical navigation systemvia the communication interface 1080 to re-calculate the imageregistration between CT or MRI image data and intraoperative surgicalfield. The communication interface 1080 may be wired or wireless.

Construct Image Mask Using Optical Properties, Thermal Properties,Tissue Properties, or Machine Learning

In another embodiment, image mask may be constructed based on tissueproperties. In one embodiment, the bony tissues and vascularized softtissues are differentiated based on optical properties. In one aspect,the optical properties may be obtained using color imaging. Forinstance, bones tend to have different color compared to muscles orligaments. Based on the color of tissues, bones and soft tissues may bedifferentiated. In another aspect, the optical properties may beobtained using hyperspectral imaging or multispectral imaging. Thehyperspectral imaging data or multispectral imaging data can revealdifferent tissue types such as bones versus soft tissues.

In another aspect, the optical properties may be obtained using infraredreflectance imaging. The infrared reflectance imaging can revealdifferent tissue types, such as bones versus soft tissues. In anotherexample, optical properties of tissues may be obtained usingtransmission mode optical imaging. In yet another example, opticalproperties of tissues such as absorption and scattering coefficient maybe obtained using optical imaging. In yet another example, opticalproperties of tissues such as oxygen saturation may be obtained usingoptical imaging. In yet another example, optical properties of tissuessuch as polarization properties may be obtained using optical imaging.Based on optical properties of tissues, an image mask may be constructedto filter certain type of tissues (e.g. soft tissue) from theintraoperative 3D scan data. For example, an image mask may beconstructed based on optical properties to filter soft tissue from theintraoperative 3D scan data, leaving only data from vertebral bodies.The filtered 3D scan data may be used for image registration andnavigation, improving the registration accuracy. In another example, animage mask may be constructed based on optical properties to filtersurgical tools/instruments from the intraoperative 3D scan data, leavingonly data from biological tissues.

The imaging detector 180 may acquire an image of optical properties. Thecontroller 110 may assign pixels or voxels of optical property type 1 topassband (logical level 1). The controller 110 may assign pixels orvoxels of optical property type 2 to rejection band (logical level 0).Controller 110 may output an image mask based on the optical properties.In one aspect, the image mask may be used with 3D scan to obtain aspatially filtered image (e.g. using logical operation AND).

The controller 110 may use optical properties to generate an image maskto include the organs of interest and exclude tissues/organs not ofinterest. The 3D scanner 160 may use the image mask to acquire anintraoperative 3D scan of the surgical field (filtered 3D scan). Thecontroller 110 may load medical image data (e.g., CT or MRI). Thecontroller 110 may perform image registration between filteredintraoperative 3D scan and medical image data. The controller 110 mayuse optical properties to generate an image mask to include the organsof interest and exclude tissues/organs not of interest. The 3D scannermay use the image mask to acquire an intraoperative 3D scan of thesurgical field (filtered 3D scan). The controller 110 may load medicalimage data (e.g., CT or MRI). The controller 110 may performsegmentation medical image data (e.g., CT or MRI) to isolate organs ofinterest (e.g., L5 of lumber spine for spine surgery). The controller110 may reduce the image data into surface data of organ of interest(e.g. surface of L5 of lumber spine). The controller 110 may use surfacedata of the organ of interest to perform surface-based imageregistration with intraoperative 3D scan.

The controller 110 may use optical properties to generate an image maskto include the organs of interest and exclude tissues/organs not ofinterest. The 3D scanner 160 may use the image mask to acquire anintraoperative 3D scan of the surgical field (filtered 3D scan). Thecontroller 110 may load medical image data (e.g., CT or MRI). Thecontroller 110 may perform segmentation on medical image data (e.g., CTor MRI) to isolate organs of interest (e.g., L5 of lumber spine forspine surgery). The controller 110 may reduce the image data intosurface data of the organ of interest (e.g., surface of L5 of lumberspine). The controller 110 may use surface data of the organ of interestto perform surface-based image registration with the intraoperative 3Dscan. The tracker 130 may track positions of at least one entity insurgery. The display 120 may display surgical navigation information tothe user.

In another embodiment, image mask may be constructed based on thermalproperties. In one aspect thermal properties may be obtained usingthermal imaging. In one embodiment, the biological tissues and surgicalinstruments/tools are differentiated based on thermal properties. Forinstance, biological tissues tend to have higher temperature compared tosurgical instruments and tools. Based on the thermal properties, tissuesand metal/plastics/tools may be differentiated. In another aspect, thethermal properties may be obtained using infrared imaging. Based onthermal properties of tissues, an image mask may be constructed tofilter surgical tools or instruments (e.g. retractor) from theintraoperative 3D scan data. For example, an image mask may beconstructed to filter surgical tools from the intraoperative 3D scandata, leaving only data from biological tissues. The filtered 3D scandata may be used for image registration and navigation, improving theregistration accuracy.

The imaging detector 180 may acquire an image of thermal properties. Thecontroller 110 may assign pixels or voxels of thermal property type 1 topassband (logical level 1). The controller 110 may assign pixels orvoxels of thermal property type 2 to rejection band (logical level 0).The controller 110 may output an image mask based on thermal properties.The controller 110 may use thermal properties to generate an image maskto include the organs of interest and exclude surgicaltools/instruments. The 3D scanner 160 may use the image mask to acquirean intraoperative 3D scan of the surgical field (filtered 3D scan). Thecontroller 110 may load medical image data (e.g., CT or MRI). Thecontroller 110 may perform image registration between filteredintraoperative 3D scan and medical image data.

The controller 110 may use thermal properties to generate an image maskto include the organs of interest and exclude surgicaltools/instruments. The 3D scanner 160 may use the image mask to acquirean intraoperative 3D scan of the surgical field (filtered 3D scan). Thecontroller 110 may load medical image data (e.g., CT or MRI). Thecontroller 110 may perform segmentation medical image data (e.g., CT orMRI) to isolate organs of interest (e.g., L5 of lumber spine for spinesurgery.) The controller 110 may reduce the image data into surface dataof the organ of interest (e.g., surface of L5 of lumber spine). Thecontroller 110 may use surface data of the organ of interest to performsurface-based image registration with intraoperative 3D scan. Thetracker 130 may track positions of at least one entity in surgery. Thedisplay 120 may display surgical navigation information to the user.

In another embodiment, image mask may be constructed based on tissueproperties obtained from other imaging modalities. In one aspect tissuesproperties may be obtained using ultrasound imaging. For example,ultrasound may be used to differentiate soft tissues from bony tissues.Either 2D ultrasound or 3D ultrasound may be used. In one example, themask may be constructed using tissue properties. The imaging detector180 may acquire an image of tissue properties (e.g., ultrasound image).The controller 110 may assign pixels or voxels of tissue property type 1to passband (logical level 1). The controller 110 may assign pixels orvoxels of tissue property type 2 to rejection band (logical level 0).The controller 110 may output an image mask based on tissue properties.

The controller 110 may use tissue properties to generate an image maskto include the organs of interest and exclude tissues/organs not ofinterest. The 3D scanner 160 may use the image mask to acquire anintraoperative 3D scan of the surgical field (filtered 3D scan). Thecontroller 110 may load medical image data (e.g., CT or MRI). Thecontroller 110 may perform image registration between filteredintraoperative 3D scan and medical image data. The controller 110 mayuse tissue properties to generate an image mask to include the organs ofinterest and exclude tissues/organs not of interest. The 3D scanner 160may use the image mask to acquire an intraoperative 3D scan of thesurgical field (filtered 3D scan). The controller 110 may load medicalimage data (e.g., CT or MRI). The controller 110 may performsegmentation medical image data (e.g., CT or MRI) to isolate the organof interest (e.g., L5 of lumber spine for spine surgery). The controller110 may reduce the image data into surface data of the organ of interest(e.g., surface of L5 of lumber spine). The controller 110 may usesurface data of the organ of interest to perform surface-based imageregistration with intraoperative 3D scan. The tracker 130 may trackpositions of the at least one entity in surgery. The display 120 maydisplay surgical navigation information to the user.

In another embodiment, image mask may be constructed based on machinelearning and image recognition. In one aspect, tissues properties may beobtained using supervised machine learning. In another aspect, tissuesproperties may be obtained using unsupervised machine learning. In yetanother aspect, tissues properties may be obtained using reinforcementlearning. In yet another aspect, tissues properties may be obtainedusing artificial neural network. For example, machine learning and imagerecognition may be used to differentiate soft tissues from bony tissues.The controller 110 may perform machine learning for tissueclassification to generate tissue class 1 and tissue class 2. Thecontroller 110 may assign pixels or voxels of tissue class 1 to passband(logical level 1). The controller 110 may assign pixels or voxels oftissue class 2 to rejection band (logical level 0). The controller 110may output an image mask based on machine learning.

The controller 110 may use machine learning to generate an image mask toinclude the organs of interest and exclude tissues/organs of interest.The 3D scanner 1607 may use the image mask to acquire an intraoperative3D scan of the surgical field (filtered 3D scan). The controller 110 mayload medical image data (e.g., CT or MRI). The controller 110 mayperform image registration between filtered intraoperative 3D scan andmedical image data. The controller 110 may use machine learning togenerate an image mask to include the organs of interest and excludetissues/organs not of interest. The 3D scanner 160 may use the imagebask to acquire an intraoperative 3D scan of the surgical field(filtered 3D scan). The controller 110 may load medical image data(e.g., CT or MRI). The controller 110 may perform segmentation medicalimage data (e.g., CT or MRI) to isolate the organ of interest (e.g., L5of lumber spine for spine surgery). The controller 110 may reduce theimage data into surface data of organ of interest (e.g., surface of L5of lumber spine). The controller 110 may use surface data of the organof interest to perform surface-based image registration withintraoperative 3D scan. The tracker 130 may track positons of at leastone entity in surgery. The display 120 may display surgical navigationinformation to the user.

Integration with Other Surgical Navigation and Robotic Surgery Systems

In one embodiment, a surgical imaging and navigation system FIG. 10comprises a controller 1010, a 3D scanning controller 150, a 3D scanner160, an intraoperative imaging controller 170, an imaging detector 180,and communication interface 1080. The 3D scanning controller 150controls the modes and properties of 3D scanner 160. For instance, thesize of the area of 3D scanning, the resolution of 3D scanning, thespeed of 3D scanning, the timing of 3D scanning may be controlled by the3D scanning controller 150. The intraoperative imaging controller 170controls the modes and properties of imaging detector 180. For instance,the size of the area of intraoperative imaging, the resolution ofintraoperative imaging, the speed of intraoperative imaging, the timingof intraoperative imaging, and the mode of intraoperative imaging may becontrolled by the intraoperative imaging controller 170. Thecommunication interface 1080 communicates with other surgical navigationsystems. In one example, the image registration calculated using thesurgical imaging and navigation system 1000 may be communicated toanother surgical navigation system via the communication interface 1080.The communication interface 1080 may be either wired or wireless. Thecontroller 1010 is in in operative communication with the 3D scanningcontroller 150, intraoperative imaging controller 170, and thecommunication interface 1080. The controller 1010 can run software suchas image registration software or computer vision algorithms to enablesurgical navigation and communicate the image registration to anothersurgical navigation system. Other surgical navigation system can havedifferent functionalities, such as intraoperative CT scan, 3Dfluoroscopy, optical tracking, or electromagnetic tracking, etc.

With an exemplary system previously discussed, a method for surgicalimaging and navigation may be implemented to provide intraoperativeguidance to surgeons and other medical professionals. The 3D scanner 160may capture a 3D scan of anatomy of a patient. The controller 1010 mayload medical image data (e.g., CT or MRI). The controller 1010 mayperform segmentation on medical image data (e.g., CT or MRI) to isolateorgan of interest (e.g., L5 of lumber spine for spine surgery). Thecontroller 1010 may reduce the image data into surface data of organ ofinterest (e.g., surface of L5 of lumber spine). The controller 1010 mayperform surface-based image registration between the surface data of theorgan of interest and the intraoperative 3D scan. The communicationsinterface 1080 may communicate the image registration to anothersurgical navigation system.

CONCLUSION

It is to be appreciated that the Detailed Description section, and notthe Abstract section, is intended to be used to interpret the claims.The Abstract section may set forth one or more, but not all exemplaryembodiments, of the present disclosure, and thus, is not intended tolimit the present disclosure and the appended claims in any way.

The present disclosure has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries may be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

It will be apparent to those skilled in the relevant art(s) the variouschanges in form and detail may be made without departing from the spirtand scope of the present disclosure. Thus the present disclosure shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A system for executing a three-dimensional (3D)intraoperative scan of a patient to generate a plurality ofintraoperative images of the patient that enables a surgeon to navigateduring a surgical operation on the patient, comprising: a 3D scannerthat includes a first image sensor and a second image sensor and isconfigured to capture a first two-dimensional (2D) intraoperative imageof a plurality of object points associated with the patient via thefirst image sensor and a second 2D intraoperative image of the pluralityof object points via the second image sensor; a 3D scanning controllerthat is configured to: project the plurality of object points includedin the first 2D intraoperative image onto a first image plane associatedwith the first image sensor and the plurality of object points includedin the second 2D intraoperative image onto a second image planeassociated with the second image sensor, determine a plurality of firstepipolar lines associated with the first image plane and a plurality ofsecond epipolar lines associated with the second image plane based on anepipolar plane that triangulates the plurality of object points includedin the first 2D intraoperative image to the plurality of object pointsincluded in the second 2D intraoperative image, wherein each epipolarline provides a depth of each object point as projected onto the firstimage plane associated with the first image sensor and the second imageplane associated with the second image sensor, and convert the first 2Dintraoperative image and the second 2D intraoperative image to the 3Dintraoperative scan of the patient based on the depth of each objectpoint provided by each corresponding epipolar line; a controller that isconfigured to: co-register pre-operative image data captured from atleast one pre-operative image of the patient with intraoperative imagedata provided by the 3D intraoperative scan, and instruct a display todisplay the co-registered pre-operative image data as captured from theat least one pre-operative image with the intraoperative image dataprovided by the 3D intraoperative scan as the surgeon navigates duringthe surgical operation.
 2. The system of claim 1, wherein the 3Dscanning controller is further configured to: generate the plurality offirst epipolar lines positioned in the first image plane of the first 2Dintraoperative image, wherein each of the first epipolar lines isparallel to each other first epipolar line as positioned in the firstimage plane; generate the plurality of second epipolar lines positionedin the second image plane of the second 2D intraoperative image, whereineach of the second epipolar lines is parallel to each other secondepipolar line as positioned in the second image plane; and convert thefirst 2D intraoperative image and the second 2D intraoperative image tothe 3D intraoperative scan of the patient based on the depth of eachobject point provided by each corresponding first epipolar line andsecond epipolar line as parallel to each other as positioned in thecorresponding first image plane and second image plane.
 3. The system ofclaim 2, wherein the 3D scanning controller is further configured to:conjugate each first epipolar line positioned in the first image planeof the first 2D intraoperative image to each corresponding secondepipolar line positioned in the second image plane of the second 2Dintraoperative image; and convert the first 2D intraoperative image andthe second 2D intraoperative image to the 3D intraoperative scan of thepatient based on the depth of each object point provided by eachcorresponding conjugate of each other as positioned in the correspondingfirst image plane and second image plane.
 4. The system of claim 3,wherein the 3D scanning controller is further configured to: generateeach first epipolar line positioned in the first image plane of thefirst 2D intraoperative image to correspond to a set of first pixelsincluded in the first 2D intraoperative image; generate each secondepipolar line positioned in the second image plane of the second 2Dintraoperative image to correspond to a set of second pixels included inthe second 2D intraoperative image; and convert the first 2Dintraoperative image and the second 2D intraoperative image to the 3Dintraoperative scan of the patient based on the depth of each set offirst pixels for each corresponding first epipolar line and the depth ofeach set of second pixels for each corresponding second epipolar line aspositioned in the first image plane and the second image plane.
 5. Thesystem of claim 4, wherein the 3D scanning controller is furtherconfigured to: generate each first epipolar line positioned in the firstimage plane of the first 2D intraoperative image to correspond to a rowof first pixels included in the first 2D intraoperative image; generateeach second epipolar line positioned in the second image plane of thesecond 2D intraoperative image to correspond to a row of second pixelsincluded in the second 2D intraoperative image; convert the first 2Dintraoperative image and the second 2D intraoperative image to the 3Dintraoperative scan of the patient based on the depth of each row offirst pixels for each corresponding first epipolar line and the depth ofeach row of second pixels for each corresponding second epipolar line aspositioned in the first image plane and the second image plane.
 6. Thesystem of claim 3, wherein the 3D scanning controller is furtherconfigured to: conduct a one-dimensional (1D) search for a correspondingpair of object points on the first epipolar line in the first imageplane of the first 2D intraoperative image and the second epipolar linein the second image plane of the second 2D intraoperative image, whereina first object point positioned on the first epipolar line correspondsto a second object point positioned on the second epipolar line; convertthe 1D search of the corresponding pair of object points on the firstepipolar line and the second epipolar line to the 3D intraoperative scanof the patient based on the depth of first object point on the firstepipolar line and the corresponding second object point on the secondepipolar line as positioned in the first image plane and the secondimage plane.
 7. The system of claim 1, wherein the first image sensor isa camera and the second image sensor is a projector.
 8. A system forexecuting a three-dimensional (3D) intraoperative scan of a patient togenerate a plurality of intraoperative images of the patient thatenables a surgeon to navigate during a surgical operation on thepatient, comprising: a 3D scanner that includes a projector, an imagesensor, and a pattern generator, wherein: the pattern generator isconfigured to generate a pseudo random pattern that includes a pluralityof dots, wherein each position of each corresponding dot included in thepseuedo random pattern is pre-determined by the pattern generator, theprojector is configured to project the pseudo random pattern onto thepatient, wherein each position of each corresponding dot included in thepseudo random pattern is projected on a corresponding position on thepatient, the image sensor is configured to capture a two-dimensional(2D) intraoperative image of a plurality of object points associatedwith the patient; a 3D scanning controller that is configured to:associate each object point associated with the patient that is capturedby the image sensor with a corresponding dot included in the pseudorandom pattern that is projected onto the patient by the projector basedon the position of each corresponding dot as pre-determined by thepattern generator, and convert the 2D intraoperative image to the 3Dintraoperative scan of the patient based on the association of eachobject point to each position of each corresponding dot included in thepseudo random pattern as pre-determined by the pattern generator; and acontroller that is configured to: co-register pre-operative image datacaptured from at least one pre-operative image of the patient withintraoperative image data provided by the 3D intraoperative scan, andinstruct a display to display the co-registered pre-operative image dataas captured from the at least one pre-operative image with theintraoperative image data provided by the 3D intraoperative scan as thesurgeon navigates during the surgical operation.
 9. The system of claim8, wherein the projector is further configured to project the pseudorandom pattern onto a 2D surface before the patient is positioned on the2D surface.
 10. The system of claim 9, wherein the image sensor isfurther configured to capture a 2D image of the pseudo random pattern asprojected onto the 2D surface before the patient is positioned on the 2Dsurface.
 11. The system of claim 10, wherein the 3D scanning controlleris further configured to: calibrate each position of each dot includedin the pseudo random pattern as projected onto the 2D surface andpre-determined by the pattern generator to each corresponding positionof each dot as included in the 2D image as captured by the image sensor;compare each position of each dot included in the pseudo random patternas projected onto the 2D surface and pre-determined by the patterngenerator to each position of each dot included in the pseudo randompattern as projected onto the patient; determine each depth of eachobject point as captured in the 2D intraoperative image by the imagesensor of the patient when the projector projects the pseudo randompattern onto the patient after the calibration based on a difference indepth of each corresponding dot included in the pseudo random pattern asprojected onto the 2D surface as compared to each corresponding dotincluded the pseudo random pattern as projected onto the patient; andconvert the 2D intraoperative image to the 3D intraoperative scan of thepatient based on the depth of each object point as provided by thecalibration of the pseudo random pattern to the 2D intraoperative image.12. The system of claim 8, wherein the 3D scanning controller is furtherconfigured to: determine a plurality of first epipolar lines associatedwith a projection image plane of the projection of the pseudo randompattern and a plurality of second epipolar lines associated withassociated with 2D intraoperative image plane of the captured 2Dintraoperative image based on an epipolar plane that triangulates theplurality of object points included in the 2D intraoperative image tothe plurality of dots included in the pseudo random pattern, whereineach epipolar line provides a depth of each object point as projectedfrom the projection image plane associated with the projector and the 2Dintraoperative image plane associated with the 2D intraoperative image;and convert the 2D intraoperative image to the 3D intraoperative scan ofthe patient based on the depth of each object point provided by eachcorresponding epipolar line.
 13. The system of claim 1, wherein the 3Dscanner and the 3D scanning controller is incorporated into a hand-heldsurgical navigation device.
 14. The system of claim 1, wherein the 3Dscanning controller is further configured to change each pseudo randompattern projected onto the patient by the projector periodically toreduce residual pattern-to-depth dependence.
 15. A system for executinga three-dimensional (3D) intraoperative scan of a patient to generate aplurality of intraoperative images of the patient that enables a surgeonto navigate during a surgical operation on the patient, comprising: a 3Dscanner that includes a projector, an image sensor, and a patterngenerator, wherein: the pattern generator is configured to generate aplurality of non-statistical patterns with each non-statistical patternincluding a plurality of identified characteristics, wherein eachplurality of identified characteristics associated with eachnon-statistical pattern are different variations of each other, theprojector is configured to project each non-statistical pattern onto thepatient in series, wherein each variation in the identifiedcharacteristics of each non-statistical pattern as projected onto thepatient is adjusted based on when in the series each correspondingnon-statistical pattern is projected onto the patient, and the imagesensor is configured to capture a two-dimensional (2D) intraoperativeimage of a plurality of object points associated with the patient aftereach non-statistical pattern is projected onto the patient; a 3Dscanning controller that is configured to: identify a position of eachobject point associated with the patient that is captured by the imagesensor after each non-statistical pattern is projected onto the patient,determine an actual position of each object point after the plurality ofnon-statistical patterns is projected onto the patient based on anaverage position of each object point determined from each identifiedposition of each object point as generated after each non-statisticalpattern is projected onto the patient, and convert the 2D intraoperativeimage to the 3D intraoperative scan of the patient based on the actualposition of each object point after the plurality of statisticalpatterns is projected onto the patient; and a controller that isconfigured to: co-register pre-operative image data captured from atleast one pre-operative image of the patient with intraoperative imagedata provided by the 3D intraoperative scan, and instruct a display todisplay the co-registered pre-operative image data as captured from theat least one pre-operative image with the intraoperative image dataprovided by the 3D intraoperative scan as the surgeon navigates duringthe surgical operation.
 16. The system of claim 15, wherein the patterngenerator is further configured to generate the plurality ofnon-statistical patterns with each non-statistical pattern being avariation in scale from each other non-statistical pattern that isprojected onto the patient.
 17. The system of claim 16, wherein thepattern generator is further configured to: generate a firstnon-statistical pattern that includes a stripe with a resolution that isdecreased to a resolution that the projector is capable to project andthe image sensor is capable to capture; and generate each additionalnon-statistical pattern that includes a stripe being an increasedvariation in scale from the first non-statistical pattern and eachadditional non-statistical pattern is a variation from each otheradditional non-statistical pattern in the resolution of each stripeassociated with each additional non-statistical pattern.
 18. The systemof claim 17, wherein the projector is further configured to: projecteach non-statistical pattern that varies in resolution to eachcorresponding horizontal row of pixels included in the 2D intraoperativeimage captured by the image sensor; and project each non-statisticalpattern that varies in resolution to each corresponding vertical columnof pixels included in the 2D intraoperative image captured by the imagesensor.
 19. The system of claim 15, wherein the 3D scanning controlleris further configured to: determine each depth of each object point ascaptured in the 2D intraoperative image by the image sensor of thepatient based on a depth associated with each pixel included in the 2Dintraoperative image that is determined after each non-statisticalpattern is projected onto the patient; and convert the 2D intraoperativeimage to the 3D intraoperative scan of the patient based on the depth ofeach object point as determined after the plurality of statisticalpatterns is projected onto the patient.
 20. The system of claim 15,wherein the 3D scanning controller is further configured to: determine aplurality of first epipolar lines associated with a projection imageplane of the projection of the plurality of non-statistical patterns anda plurality of second epipolar lines associated with a 2D intraoperativeimage plane of the captured 2D intraoperative image based on an epipolarplane that triangulates the plurality of object points generated wheneach non-statistical pattern is applied to the 2D intraoperative imageto the plurality of object points included in the 2D intraoperative,wherein each epipolar line provides a depth of each object point asprojected from the projection image plane associated with the projectorand the 2D intraoperative image plane associated with the 2Dintraoperative image; and convert the 2D intraoperative image to the 3Dintraoperative scan of the patient based on the depth of each objectpoint provided by each corresponding epipolar line.